scrapyd@a4c2642d74db:~/projects$ git clone https://github.com/nutthaphon/PyLearning.git Cloning into 'PyLearning'... remote: Counting objects: 323, done. remote: Compressing objects: 100% (6/6), done. remote: Total 323 (delta 0), reused 0 (delta 0), pack-reused 317 Receiving objects: 100% (323/323), 61.27 KiB | 0 bytes/s, done. Resolving deltas: 100% (146/146), done. Checking connectivity... done. scrapyd@a4c2642d74db:~/projects$ scrapyd@a4c2642d74db:~/projects$ scrapyd@a4c2642d74db:~/projects$ scrapyd@a4c2642d74db:~/projects$ cd PyLearning/ scrapyd@a4c2642d74db:~/projects/PyLearning$ ls Animals CherryPy DJango README.md Scraping dataset decor foo serial test tutorial scrapyd@a4c2642d74db:~/projects/PyLearning$ cd dataset/ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ ls __init__.py __init__.pyc build project.egg-info scrapinghub.yml scrapy.cfg settrade setup.py scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ sed -i 's/#url/url/g' scrapy.cfg scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ cat scrapy.cfg # Automatically created by: scrapy startproject # # For more information about the [deploy] section see: # https://scrapyd.readthedocs.org/en/latest/deploy.html [settings] default = settrade.settings [deploy] url = http://localhost:6800/ project = settrade scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ ls __init__.py __init__.pyc build project.egg-info scrapinghub.yml scrapy.cfg settrade setup.py scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ cd settrade/ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset/settrade$ ls DailyStockQuote.py QueryStockSymbol.py ThinkSpeakChannels.py __init__.py items.py scrapinghub.yml settings.pyc DailyStockQuote.pyc QueryStockSymbol.pyc ThinkSpeakChannels.pyc __init__.pyc pipelines.py settings.py spiders scrapyd@a4c2642d74db:~/projects/PyLearning/dataset/settrade$ cd spiders/ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset/settrade/spiders$ ls SettradeSpider.py SettradeSpider.pyc __init__.py __init__.pyc scrapyd@a4c2642d74db:~/projects/PyLearning/dataset/settrade/spiders$ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset/settrade/spiders$ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset/settrade/spiders$ ls SettradeSpider.py SettradeSpider.pyc __init__.py __init__.pyc scrapyd@a4c2642d74db:~/projects/PyLearning/dataset/settrade/spiders$ cd .. scrapyd@a4c2642d74db:~/projects/PyLearning/dataset/settrade$ cd .. scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ ls __init__.py __init__.pyc build project.egg-info scrapinghub.yml scrapy.cfg settrade setup.py scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ ~/scrapyd-client-master/scrapyd-client/scrapyd-deploy -l default http://localhost:6800/ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ ~/scrapyd-client-master/scrapyd-client/scrapyd-deploy default -p dataset Packing version 1479804720 Deploying to project "dataset" in http://localhost:6800/addversion.json Server response (200): {"status": "ok", "project": "dataset", "version": "1479804720", "spiders": 1, "node_name": "a4c2642d74db"} scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ ~/scrapyd-client-master/scrapyd-client/scrapyd-deploy -L default tutorial dataset scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ curl http://localhost:6800/listprojects.json {"status": "ok", "projects": ["tutorial", "dataset"], "node_name": "a4c2642d74db"} scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ curl http://localhost:6800/listspiders.json?project=dataset {"status": "ok", "spiders": ["settrade_dataset"], "node_name": "a4c2642d74db"} scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ scrapyd@a4c2642d74db:~/projects/PyLearning/dataset$ curl http://localhost:6800/schedule.json -d project=dataset -d spider=settrade_dataset -d setting=DOWNLOAD_DELAY=5 -d start_urls=http://www.settrade.com/servlet/IntradayStockChartDataServlet?symbol=INTUCH,http://www.settrade.com/servlet/IntradayStockChartDataServlet?symbol=CPF,http://www.settrade.com/servlet/IntradayStockChartDataServlet?symbol=ADVANC {"status": "ok", "jobid": "c60e4566b09111e68c380242ac110002", "node_name": "a4c2642d74db"}
Tuesday, November 22, 2016
Deploy dataset project to scrapyd
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment