大表数据以CSV格式导出之PHP
2019-11-01 本文已影响0人
胡乱唱歌ing
说明:工作中经常会遇到需要把用户表、日志表以excel的形式导出,如果表的的数据小的话还好操作,但是表足够大的话,很容易出现内存溢出、脚本超时的问题。比较好的解决方案就是php+redis队列任务实现大表导出,以下是php脚本处理队列任务的具体表现。
思路:1.不限制脚本超时时间。2.把数据一页一页的写入文件,这样就避免了内存溢出的情况
public function actionExportToCsv(string $table_name)
{
set_time_limit(0);
echo "export ". $table_name . "start ....";
echo "\n";
$sql = " SELECT count(*) as num FROM {$table_name}";
$count = Yii::$app->db->createCommand($sql)->queryScalar();
$headArr = ['id','iTime','sInputPort','sOutPort','sSourceAddr','sSourcePort','sProtocol','sTargetAddr','sTargetPort','sAction','iCount'];
$path =Yii::getAlias("@console").'/runtime/data/'.$table_name.'_'.time().'.csv';
$fp = fopen($path, "w") or die("Unable to open file!\n");
fwrite($fp, chr(0xEF).chr(0xBB).chr(0xBF));
fputcsv($fp,$headArr);
if($count)
{
$limit = 100; //每页导100条数据
$page = 1;
$offset = 0;
while ($offset <= $count) {
//获取每一页的数据并写入文件
$offset = ($page-1) * $limit;
$sql = "SELECT * FROM {$table_name} ORDER BY id DESC LIMIT ".$offset.','.$limit;
$data = Yii::$app->db->createCommand($sql)->queryAll();
if(!$data)
{
break;
}
foreach ($data as $value) {
fputcsv($fp,$value);
}
$page++; //翻页
}
}
fclose($fp);
echo $path ."\n";
return ;
}