PRESTASHOP background/cron CSV products import (modified)
I have modified the code of https://coderwall.com/p/2q5lcw by https://coderwall.com/croustibat
file: ps_integrator.php
/* ORIGINAL CODE
https://coderwall.com/p/2q5lcw
by https://coderwall.com/croustibat
modified by Simone "Magicianred" Paolucci
http://simone.paolucci.name */
require_once ('class.PSRequest.php'); //see class source code below
/* START Configuration */
$shopID = "myfirstshop"; // ID or unqiue name of the target shop
$pathToWriteFile = __DIR__."/";
$adminUrl = 'http://prestashop.dev:8080/admin6596/';
$adminLoginEmail = YOUR_PRESTASHOP_EMAIL;
$adminLoginPass = YOUR_PRESTASHOP_PASS;
/* END Configuration */
ob_start();
// your set of product object or array
$products = array();
$products[] = array('id'=>1,'name'=>'Giostra a pedali','reference'=>'prodotto1');
$products[] = array('id'=>2,'name'=>'Pedalò a vela','reference'=>'prodotto2');
$products[] = array('id'=>3,'name'=>'Idrovolante subaqueo','reference'=>'prodotto3');
$products[] = array('id'=>4,'name'=>'Hovercraft spaziale','reference'=>'prodotto4');
$products[] = array('id'=>5,'name'=>'Trottola stazionaria','reference'=>'prodotto5');
$products[] = array('id'=>15,'name'=>'Trottola stazionaria','reference'=>'prodotto15');
$products[] = array('id'=>25,'name'=>'Trottola stazionaria','reference'=>'prodotto25');
$products[] = array('id'=>35,'name'=>'Trottola stazionaria','reference'=>'prodotto35');
// CSV first line for products
//$csv = "id;Active (0/1);Name*;Categories (x,y,z,...);Price tax excluded;Tax rules id;Wholesale price;On sale (0/1);Discount amount;Discount percent;Discount from (yyy-mm-dd);Discount to (yyy-mm-dd);Reference #;Supplier reference #;Supplier;Manufacturer;EAN13;UPC;Ecotax;Weight;Quantity;Short description;Description;Tags (x,y,z,...);Meta-title;Meta-keywords;Meta-description;URL rewritten;Text when in-stock;Text if back-order allowed;Available for order (0 = No, 1 = Yes);Product creation date;Show price (0 = No, 1 = Yes);Image URLs (x,y,z,...);Delete existing images (0 = No, 1 = Yes);Feature (Name:Value:Position);Available online only (0 = No, 1 = Yes);Condition (new,used,refurbished);ID / Name of shop".PHP_EOL;
$csv = "id;Active (0/1);Name*;reference;price;barcode".PHP_EOL;
foreach ($products as $prod) {
// fill in the csv string with your datas
$csv .= $prod['id'].";0;".html_entity_decode($prod['name'])."CCCC;".$prod['reference'].';1;1234567890123434'.PHP_EOL;
}
$now=time();
$csvname = $now."-PRODUCTS-".$shopID.".csv";
$file = $pathToWriteFile.$csvname;
file_put_contents($file, $csv);
echo "Write file : ".$file.PHP_EOL;
echo "Login on Prestashop Admin area...".PHP_EOL;
$request = new PSRequest();
$request->setCookiFileLocation(__DIR__.'/PScookie.txt');
$request->setPost(array("email" => $adminLoginEmail,"passwd" => $adminLoginPass, "submitLogin" => "Connexion")); // you must be a super admin
$request->call($adminUrl."index.php?controller=AdminLogin");
echo "Get token...".PHP_EOL;
$request->call($adminUrl."index.php?controller=AdminImport");
list(,$response) = explode(PHP_EOL.PHP_EOL, $request->_webpage, 2);
preg_match("/token=([a-z0-9]+)/", $response, $matches);
$token = $matches[1];
echo "Token : ".$token.PHP_EOL;
// START Carico il file tramite AJAX
srand((double)microtime()*1000000);
$rand = rand(0,9999999999999999);
$rand = str_pad($rand, 14, "0", STR_PAD_BOTH);
$handle = fopen($file, "r");
$fileSize = filesize($file);
$fileContent = fread($handle, $fileSize);
fclose($handle);
$request->setFileToUpload($file,$csvname,'application/csv');
$request->call($adminUrl."index.php?controller=AdminImport&token=".$token."&ajax=1&action=uploadCsv&rand=".$rand);
list(,$response) = explode(PHP_EOL.PHP_EOL, $request->_webpage, 2);
preg_match('/"filename":"(.*?)"/', $response, $matches);
$returnFilename = $matches[1];
// END Carico il file tramite AJAX
unset($request);
$request = new PSRequest();
$request->setCookiFileLocation(__DIR__.'/PScookie.txt');
// Send POST datas just like the admin form would do it, those datas depends on what you want to do : check the import admin page.
$request->setPost(array(
"controller" => "AdminImport",
"token" => $token,
"skip" => 1,
"csv" => $returnFilename,
"convert" => '',
"regenerate" => '',
"entity" => 1, //1 is for products import
"iso_lang" => "it",
"truncate" => 1,
"forceIDs" => 1,
"match_ref" => 0,
"separator" => ";",
"multiple_value_separator" => ",",
"import" => 1,
"type_value" => array( 0 => 'id', 1 => 'active', 2 => 'name', 3 => 'reference', 4 => 'price_tex', 5 => 'ean13')
//"type_value" => array( 0 => 'id', 1 => 'active', 2 => 'name', 3 => 'category', 4 => 'price_tex', 5 => 'id_tax_rules_group', 6 => 'wholesale_price', 7 => 'on_sale', 8 => 'reduction_price', 9 => 'reduction_percent', 10 => 'reduction_from', 11 => 'reduction_to', 12 => 'reference', 13 => 'supplier_reference', 14 => 'supplier', 15 => 'manufacturer', 16 => 'ean13', 17 => 'upc', 18 => 'ecotax', 19 => 'weight', 20 => 'quantity', 21 => 'description_short', 22 => 'description', 23 => 'tags', 24 => 'meta_title', 25 => 'meta_keywords', 26 => 'meta_description', 27 => 'link_rewrite', 28 => 'available_now', 29 => 'available_later', 30 => 'available_for_order', 31 => 'date_add', 32 => 'show_price', 33 => 'image', 34 => 'delete_existing_images', 35 => 'features', 36 => 'online_only', 37 => 'condition', 38 => 'shop')
)
);
echo "call AdminImport and POST datas...".PHP_EOL;
$returnDanger = ''; $returnWarning = '';
$request->call($adminUrl."index.php?controller=AdminImport&token=".$token);
list(,$response) = explode(PHP_EOL.PHP_EOL, $request->_webpage, 2);
// recupero gli warning
$warning_pattern = '/\<div class="alert alert-warning">([\s\S]*)\<\/div>/';
preg_match($warning_pattern, $response, $warning_matches);
if(isset($warning_matches[1])) $returnWarning = $warning_matches[1];
if(strpos($returnWarning,"<h4>")) { $returnWarning = substr($returnWarning,strpos($returnWarning,"<h4>")-4,strlen($returnWarning)); }
if(strpos($returnWarning,"</div>")) { $returnWarning = substr($returnWarning,0,strpos($returnWarning,"</div>",2)); }
// recupero gli errori
$danger_pattern = '/\<div class="alert alert-danger">([\s\S]*)\<\/div>/';
preg_match($danger_pattern, $response, $danger_matches);
if(isset($danger_matches[1])) $returnDanger = $danger_matches[1];
if(strpos($returnDanger,"<h4>")) { $returnDanger = substr($returnDanger,strpos($returnDanger,"<h4>")-4,strlen($returnDanger)); }
if(strpos($returnDanger,"</div>")) { $returnDanger = substr($returnDanger,0,strpos($returnDanger,"</div>",2)); }
$returnResponse = $returnDanger . $returnWarning;
//echo PHP_EOL.PHP_EOL."RESPONSE: ".PHP_EOL.PHP_EOL. $returnResponse;
echo $request->_webpage;
echo "-- END --".PHP_EOL;
$request = null;
$content = ob_get_contents();
ob_end_clean();
$contentHtml = str_replace(PHP_EOL,'<br>',$content);
echo $content;
$f = fopen($now."-log-import.txt", "w");
fwrite($f, $content);
fclose($f);
file: class.PSRequest.php
/* ORIGINAL CODE
https://coderwall.com/p/2q5lcw
by https://coderwall.com/croustibat
modified by Simone "Magicianred" Paolucci
http://simone.paolucci.name
*/
class PSRequest {
protected $_eol = "\r\n";
protected $_useragent = 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2';
protected $_cookieFileLocation = './cookie.txt';
protected $_referer = "http://www.google.com";
protected $_url;
protected $_followlocation;
protected $_timeout;
protected $_maxRedirects;
protected $_post = false;
protected $_multipart = false;
protected $_file = false;
protected $_postFields;
protected $_postFile;
protected $_session;
protected $_includeHeader;
protected $_noBody;
protected $_status;
protected $_binaryTransfer;
protected $_file_to_upload = null;
protected $_file_to_upload_size = 0;
protected $_file_name = '';
protected $_file_transfer_codebase = false;
protected $_file_content_type = '';
protected $_boundary = 'boundaryAAAbbb';
public $_webpage;
public $authentication = 0;
public $auth_name = '';
public $auth_pass = '';
protected $ch; // curl handler
public function __construct($url = '', $followlocation = true, $timeOut = 30, $maxRedirecs = 4, $binaryTransfer = false, $includeHeader = true, $noBody = false)
{
$this->_url = $url;
$this->_followlocation = $followlocation;
$this->_timeout = $timeOut;
$this->_maxRedirects = $maxRedirecs;
$this->_noBody = $noBody;
$this->_includeHeader = $includeHeader;
$this->_binaryTransfer = $binaryTransfer;
$this->_cookieFileLocation = dirname(__FILE__).'/cookie.txt';
$this->ch = curl_init();
}
public function __destruct() {
curl_close($this->ch);
}
public function useAuth($use){
$this->authentication = 0;
if($use == true) $this->authentication = 1;
}
public function setEndOfLine($chars) {
$this->_eol = $chars;
}
public function setName($name){
$this->auth_name = $name;
}
public function setPass($pass){
$this->auth_pass = $pass;
}
public function setBoundary($boundary) {
$this->_boundary = $boundary;
}
public function setReferer($referer){
$this->_referer = $referer;
}
public function setCookiFileLocation($path)
{
$this->_cookieFileLocation = $path;
}
public function setFileToUpload($filePath, $filename, $contentType='plain/text')
{
$this->setPostMultipart(array('post'=>'true'));
$this->_file = true;
$this->_file_name = $filename;
$this->_file_content_type = $contentType;
//$this->_file_to_upload = fopen($filePath,'r');
$handle = fopen($filePath, "r");
$this->_file_to_upload_size = filesize($filePath);
$this->_file_to_upload = fread($handle, $this->_file_to_upload_size);
fclose($handle);
}
public function setPostMultipart($postFields)
{
$this->_post = true;
$this->_multipart = true;
if (is_array($postFields)) {
$fields_string = $this->multipart_build_query($postFields);
}
else {
$fields_string = $postFields;
}
$this->_postFields = $fields_string;
}
public function setPost($postFields)
{
$this->_post = true;
if (is_array($postFields)) {
$fields_string = http_build_query($postFields);
}
else {
$fields_string = $postFields;
}
$this->_postFields = $fields_string;
}
public function setUserAgent($userAgent)
{
$this->_useragent = $userAgent;
}
public function call($url = null, $header = null)
{
if(is_null($header)) {
if($this->_multipart == true) {
$header = array("Content-Type: multipart/form-data; boundary=".$this->_boundary);
} else {
$header = array('Content-Type: application/x-www-form-urlencoded');
}
}
if ($url) {
$this->_url = $url;
}
if (!$url) {
throw new Exception('You should set an URL to call.');
}
curl_setopt($this->ch,CURLOPT_URL,$this->_url);
curl_setopt($this->ch,CURLOPT_HTTPHEADER, $header);
curl_setopt($this->ch,CURLOPT_TIMEOUT,$this->_timeout);
curl_setopt($this->ch,CURLOPT_MAXREDIRS,$this->_maxRedirects);
curl_setopt($this->ch,CURLOPT_RETURNTRANSFER,true);
curl_setopt($this->ch,CURLOPT_FOLLOWLOCATION,$this->_followlocation);
curl_setopt($this->ch,CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($this->ch,CURLOPT_COOKIESESSION, true );
curl_setopt($this->ch,CURLOPT_COOKIEJAR,$this->_cookieFileLocation);
curl_setopt($this->ch,CURLOPT_COOKIEFILE,$this->_cookieFileLocation);
if ($this->authentication == 1) {
curl_setopt($this->ch, CURLOPT_USERPWD, $this->auth_name.':'.$this->auth_pass);
}
if ($this->_multipart) {
curl_setopt($this->ch,CURLOPT_POST,true);
if($this->_file) {
$this->_postFields .= $this->add_multipart_build_file('file',$this->_file_name,$this->_file_content_type);
$this->_postFields .= "--".$this->_eol;
curl_setopt($this->ch,CURLOPT_INFILESIZE, $this->_file_to_upload_size);
curl_setopt($this->ch, CURLOPT_BINARYTRANSFER, 1);
}
} else if ($this->_post) {
curl_setopt($this->ch,CURLOPT_POST,true);
}
curl_setopt($this->ch,CURLOPT_POSTFIELDS,$this->_postFields);
if ($this->_includeHeader) {
curl_setopt($this->ch,CURLOPT_HEADER,true);
}
if ($this->_noBody) {
curl_setopt($this->ch,CURLOPT_NOBODY,true);
}
/* if ($this->_file_to_upload_size > 0 && !is_null($this->_file_to_upload)) {
curl_setopt($this->ch, CURLOPT_READFUNCTION, 'uploadFileCall');
} */
curl_setopt($this->ch,CURLOPT_USERAGENT,$this->_useragent);
curl_setopt($this->ch,CURLOPT_REFERER,$this->_referer);
$this->_webpage = curl_exec($this->ch);
/*if (curl_errno($this->ch)) {
print "<hr><hr>Error: ". curl_error($this->ch) ."<hr><hr>";
}*/
$this->_status = curl_getinfo($this->ch,CURLINFO_HTTP_CODE);
return $this->_webpage;
}
public function getHttpStatus()
{
return $this->_status;
}
public function __tostring(){
return $this->_webpage;
}
/*function uploadFileCall($ch, $data){
return fread($this->_file_to_upload, $this->_file_to_upload_size);
}*/
function multipart_build_query($fields){
$retval = '';
foreach($fields as $key => $value){
$retval .= "--".$this->_boundary.$this->_eol."Content-Disposition: form-data; name=\"".$key."\"".$this->_eol.$this->_eol.$value.$this->_eol;
}
//$retval .= "--". $this->_boundary ."--".$this->_eol;
$retval .= "--". $this->_boundary .$this->_eol;
return $retval;
}
function add_multipart_build_file($key,$filename='file.csv',$contentType ="application/csv") {
$retval = '';
$retval .= "Content-Disposition: form-data; name=\"$key\"; filename=\"$filename\"".$this->_eol;
$retval .= "Content-Type: $contentType ".$this->_eol.$this->_eol;
if($this->_file_transfer_codebase == true) {
$retval .= 'Content-Transfer-Encoding: base64'.$this->_eol.$this->_eol;
$retval .= chunk_split(base64_encode($this->_file_to_upload));
} else {
$retval .= $this->_file_to_upload;
}
$retval .= "--". $this->_boundary; // ."--".$this->_eol;
return $retval;
}
}
Enjoy your code.
Written by Simone Paolucci
Related protips
32 Responses
Hi, I'm trying to setup your solution, but I have some issues, the $request->_webpage (line ~64) does not return any value so no token have been retrieved.
Do you have any idea what is missing?
regards
You have to change this line with your url prestashop installation?
$adminUrl = 'http://prestashop.dev:8080/admin6596/';
You have to try to charge your url (your $adminUrl) in browser if it's ok?
Check in your apache errors log file if there are any other errors.
hi steefano974, in line 64. you should change to
preg_match("/AdminImport&token=([a-z0-9]+)/", $response, $matches);
and comment line $request->call($adminUrl."index.php?controller=AdminImport");
This solution is get token from Dashboard page for this controller.
Sorry my english.
Thank so much this artical, Simone Paolucci.
Thank you toanloau :)
Hi !
I have the exact same problem as steefano974 : my _webpage attribute is always void. My login works well, though, for I get a token from AdminLogin. I tried to use AdminLogin token to build my request, but nothing happens when calling AdminImport controller...
This is boring because I urgently need to make attributes import work for my customer. I tried toanloau's trick but it fails miserably. I can't catch any error while importing.
Any idea ? I'm in desperate need of a solution :(
I don't know. I believe there are some server specific problem.
Try to add curl verbose option in 'call' function of class.PSRequest.php file:
curlsetopt($this->ch,CURLOPTVERBOSE,true);
Also add in each file these instrunction to notify some errors:
errorreporting(-1);
iniset('display_errors', 'On');
Detect if 'allowurlfopen' is enabled:
if( iniget('allowurl_fopen') ) {
echo 'Is enabled';
}
Give me some feedback
I'm trying to use this script too.
The file is correctly uploaded but nothing happen after...
When i do it manually with the uploaded file, got products imported...
I'm on PrestaShop™ 1.6.0.8, any suggestion ? i don't see any error that can explain that...
Active error display:
errorreporting(EALL);
iniset('displayerrors', 1);
the row:
echo "Token : ".$token.PHP_EOL;
print your token?
then try to use advises in my previous post and give me your feedback
Yes, i got the token (with some modification), when i do a echo of $request->webpage
I'm logged in AdminImport.
There is a prestashop notification error about an empty product name, but there is not.
And when i try it manually, I don't have this error notification.
I verified like 5 times the typevalue and columns of the csv, can't see a problem...
I got no error.
So i d'ont understand why
$matchColumn = array(
0 => 'reference',
1 => 'ean13',
2 => 'active',
3 => 'name',
4 => 'category',
5 => 'pricetex',
1 => 'manufacturer',
2 => 'quantity',
3 => 'descriptionshort',
4 => 'description',
5 => 'metatitle',
4 => 'metadescription',
5 => 'image',
6 => 'id',
7 => 'idtaxrules_group'
);
$importType=1;
$request->setPost(array(
"controller" => "AdminImport",
"token" => $token,
"skip" => 0,
"csv" => $returnFilename,
"convert" => '',
"regenerate" => '',
"entity" => $importType, //1 is for products import
"isolang" => "fr",
"truncate" => 0,
"forceIDs" => 0,
"matchref" => 1,
"separator" => ";",
"multiplevalueseparator" => ",",
"import" => 1,
"type_value" => $matchColumn
You have check if the header of csv (the $csv = "id;Active (0/1);Name*;reference;price;barcode".PHP_EOL; variable), the lines of the csv and your $matchColumn have to match.
If the csv headers not match the rows prestashop ignore its.
Well it match, i check it few times. I opened the uploaded file with Excel to have better view, every columns is here.
$csv = "reference;ean13;actif;nom;categories;prix;marque;Quantité;Description courte;Description;Balise de titre;Méta description;URLs des images;id;id tax rule" . PHP_EOL;
Example of generated line :
198109913;;1;GUGA;BRAZBIKE;33.33;ESOL;0;;;;;http://website.com/media/photos/1-EXT1.png;198109913;3
I got the same error when i echo the webpage :
No Name (ID: ESOL) ne peut pas être enregistré
Property Product->name is empty
But the manufacturer existe, in the right shop id...
Ok i'm just stupid...
If you check my $matchColumn the key of my array elements are wrong...
My bad...
I have detected a problem with version 1.5 of Prestashop. This version does not have the file upload Ajax, so you have to copy the csv files in the folder admin/import and pass the file name to the form of importCsv. A brief update of the script.
magicianred ... Grandissimo! ora provo il tuo script!
Hi, with prestashop 1.6.0.11 I get Invalid token, but the script is compatible with this ver?
Hello magicianred! Please I suggest a short tutorial to run the script in prestashop. directory where to put and call funcioin the script. thank you very much
Notice: Undefined offset: 1 in D:\www\prestashop\at\ps_integrator.php on line 60
Notice: Undefined offset: 1 in D:\www\prestashop\at\psintegrator.php on line 62
Token :
Notice: Undefined offset: 1 in D:\www\prestashop\at\psintegrator.php on line 79
Notice: Undefined offset: 1 in D:\www\prestashop\at\psintegrator.php on line 81
call AdminImport and POST datas...
Notice: Undefined offset: 1 in D:\www\prestashop\at\psintegrator.php on line 112
-- END --
Prestashop 1.6 :C
Denwer : "Fatal error: Cannot redeclare class PSRequest in K:\home*\www\cron\class.PSRequest.php on line 10"
Web: empty screen
If I remove the line 10 (require_once ('class.PSRequest.php');), while imports of passes , but the product does not appear in the list .
Sorry for being a little description . I am Russian.
Thnx for help.
Hello. My mistake was in the line with the position of the data in the array. Thank you for the code .
And please tell me . Maybe you can somehow connect to send data from csv? Thank you.
I can't seem to get it working. It imports nothing and when the end view opens it mentions that the csv is not correct. Strange, because when I try to add manually, the csv is imported
Retested deeply and I have an import error:
{"file":{"name":"product2015-12-01220016.csv","type":"","tmp_name":"","error":"The uploaded file was only partially uploaded.","size":0}}
How to fix this? I already increased the settings in php.ini
@EricCulus: try it https://www.prestashop.com/forums/topic/270830-solved-manufacturersupplier-csv-import-logo-ps-15/
@bogdandin: Usually if the Token is empty it's a login problem. Check your credential.
"Write file : /home/f1shopby/public_html/f1shop.by/*******.csv Login on Prestashop Admin area... Get token... Token : call AdminImport and POST datas... HTTP/1.1 200 OK Server: nginx/1.8.0 Date: Mon, 04 Jan 2016 13:49:41 GMT Content-Type: text/html; charset=utf-8 Transfer-Encoding: chunked Connection: keep-alive X-Powered-By: PHP/5.3.29 Vary: Accept-Encoding,User-Agent"
Help please, Presta 1.6.1.3. Time on 16:50 04.01.2016
"str 62 | list(,$response) = explode(PHPEOL.PHPEOL, $request->webpage, 2);"
Perhaps sometime accidentally put a comma and preserved.
errorlog helped me)
great module, works fine, but how to import from multipart csv files. First file should have truncate option to "yes" and all others to "no"
Hi all
but how can I invoke the script in the BO of PrestaShop.
for those who has problem getting the token, this worked for me:
$request = new PSRequest();
$request->setCookiFileLocation(DIR.'/PScookie.txt');
$request->setPost(array("email" => $adminLoginEmail,"passwd" => $adminLoginPass, "submitLogin" => "Connexion")); // you must be a super admin
$request->call($adminUrl."index.php?controller=AdminLogin");
$request->call($adminUrl."index.php?controller=AdminImport");
pregmatch("/token=([a-z0-9]+)/", $request->webpage, $matches);
$token = $matches[1];
Hello.
Thank you very much for this code. It worked for me. I'm using prestashop 1.6.1.13
I have a question. Is it possible to split the csv file during the process of uploading it?
I saw in the "class.PSRequest.php" file there is an option called:
protected $_multipart = false;
I wonder if just changing this value to "true" would do the trick.
Thanks in advance for your answer.
How can I upload more than one csv file?
My ISP only allows me to program 1 cron job for the command /usr/bin/php...
I am trying to upload (and be updating a database of about 20,000 products)
any help will be really appreciated...
Hi magicianred I'm trying the script with prestashop 1.7.1.4 and the credential are correct.
I've a problem with the token like bogdandin.
Phperror is:
PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs//import_cron.php on line 63
PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs//importcron.php on line 70
PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs/***/importcron.php on line 89
PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs//import_cron.php on line 92
PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs//import_cron.php on line 181
Any ideas??
Thx
Matteo
I've been asked to look into this by a friend.
Is anyone else interested in using this mechanism (scripted) to update product catalogue in the latest version ( 1.7.6.9)