Creating a snapshot of a blog post using wget
May 21, 2011 in Uncategorized
In order to achieve the preservation of a blog post, one has to copy all the files that are necessary to properly display the given HTML page. We will try to achieve this using GNU Wget, a free utility for non-interactive download of files from the web. Our target will be to create a snapshot of this blog post: http://blogforever.eu/blog/2011/05/17/rendering-and-storing-web-pages-using-wkhtmltopdf/
First trial
Invoking the following command results in the creation of a single html file index.html containing the html page of the blog post.
[bash]
wget http://blogforever.eu/blog/2011/05/17/rendering-and-storing-web-pages-using-wkhtmltopdf/
[/bash]
This is not good enough as the blog post consists of many different elements such as images, css, javascript, etc that are parts of the web page and we are missing these files.
Second Trial
So, after a little research we come up with this command:
[bash]
wget -p -Ppost2 –convert-links http://blogforever.eu/blog/2011/05/17/rendering-and-storing-web-pages-using-wkhtmltopdf/
[/bash]
The extra parameters explanation:
- -p : This option causes Wget to download all the files that are necessary to properly display a given HTML page. This includes such things as inlined images, sounds, and referenced stylesheets.
- -Ppost2 : Set directory prefix. The directory prefix is the directory where all other files and subdirectories will be saved to, i.e. the top of the retrieval tree. The default is . (the current directory).
- –convert-links : After the download is complete, convert the links in the document to make them suitable for local viewing. This affects not only the visible hyperlinks, but any part of the document that links to external content, such as embedded images, links to style sheets, hyperlinks to non-HTML content, etc.
The result of this command is to retrieve all the web page’s files and create the following file structure:
[bash]
post2
post2/blogforever.eu
post2/blogforever.eu/index.php?ak_action=aktt_js&v=2.4
post2/blogforever.eu/index.php?ak_action=aktt_css&v=2.4
post2/blogforever.eu/wp-content
post2/blogforever.eu/wp-content/plugins
post2/blogforever.eu/wp-content/plugins/buddypress
post2/blogforever.eu/wp-content/plugins/buddypress/bp-themes
post2/blogforever.eu/wp-content/plugins/buddypress/bp-themes/bp-default
post2/blogforever.eu/wp-content/plugins/buddypress/bp-themes/bp-default/style.css
post2/blogforever.eu/wp-content/plugins/buddypress/bp-themes/bp-default/_inc
post2/blogforever.eu/wp-content/plugins/buddypress/bp-themes/bp-default/_inc/css
post2/blogforever.eu/wp-content/plugins/buddypress/bp-themes/bp-default/_inc/css/default.css
post2/blogforever.eu/wp-content/plugins/buddypress/bp-themes/bp-default/_inc/css/adminbar.css
post2/blogforever.eu/wp-content/plugins/buddypress/bp-themes/bp-default/_inc/global.js?ver=3.1.2
post2/blogforever.eu/wp-content/plugins/smooth-slider
post2/blogforever.eu/wp-content/plugins/smooth-slider/css
post2/blogforever.eu/wp-content/plugins/smooth-slider/css/styles
post2/blogforever.eu/wp-content/plugins/smooth-slider/css/styles/default.css?ver=2.3.2
post2/blogforever.eu/wp-content/plugins/smooth-slider/js
post2/blogforever.eu/wp-content/plugins/smooth-slider/js/stepcarousel.js?ver=2.3.2
post2/blogforever.eu/wp-content/themes
post2/blogforever.eu/wp-content/themes/academica
post2/blogforever.eu/wp-content/themes/academica/images
post2/blogforever.eu/wp-content/themes/academica/images/fp7-eu.png
post2/blogforever.eu/wp-content/uploads
post2/blogforever.eu/wp-content/uploads/2011
post2/blogforever.eu/wp-content/uploads/2011/03
post2/blogforever.eu/wp-content/uploads/2011/03/blogforever-logo.png
post2/blogforever.eu/wp-content/uploads/avatars
post2/blogforever.eu/wp-content/uploads/avatars/14
post2/blogforever.eu/wp-content/uploads/avatars/14/4cb1eb4d850a67a41f0303e627e59125-bpthumb.jpg
post2/blogforever.eu/wp-content/uploads/avatars/1
post2/blogforever.eu/wp-content/uploads/avatars/1/815cf9f0cb91ff8810647c9538dec2c2-bpthumb.jpg
post2/blogforever.eu/wp-content/uploads/avatars/25
post2/blogforever.eu/wp-content/uploads/avatars/25/2401398b6806dd6321b0e97278fd2580-bpthumb.jpg
post2/blogforever.eu/wp-content/uploads/avatars/22
post2/blogforever.eu/wp-content/uploads/avatars/22/77e6f2d2239ace354e9b276c0363dffe-bpthumb.jpg
post2/blogforever.eu/wp-content/uploads/avatars/21
post2/blogforever.eu/wp-content/uploads/avatars/21/18a1ba35c3b12649c40d2b747f5f6d1b-bpthumb.jpg
post2/blogforever.eu/wp-content/uploads/avatars/15
post2/blogforever.eu/wp-content/uploads/avatars/15/9f7efbf7f2902c832da6bd141187265f-bpthumb.jpg
post2/blogforever.eu/blog
post2/blogforever.eu/blog/2011
post2/blogforever.eu/blog/2011/05
post2/blogforever.eu/blog/2011/05/17
post2/blogforever.eu/blog/2011/05/17/rendering-and-storing-web-pages-using-wkhtmltopdf
post2/blogforever.eu/blog/2011/05/17/rendering-and-storing-web-pages-using-wkhtmltopdf/index.html
post2/blogforever.eu/robots.txt
post2/blogforever.eu/wp-includes
post2/blogforever.eu/wp-includes/js
post2/blogforever.eu/wp-includes/js/comment-reply.js?ver=20090102
post2/blogforever.eu/wp-includes/js/l10n.js?ver=20101110
post2/blogforever.eu/wp-includes/js/jquery
post2/blogforever.eu/wp-includes/js/jquery/jquery.js?ver=1.4.4
[/bash]
It is clear that this command has retrieved all web page elements but the resulting file structure is complex and inefficient as it is a replicate of the web server’s file structure! For instance, one of the problems arising is that the index.html filename containing the HTML page is located in post2/blogforever.eu/blog/2011/05/17/rendering-and-storing-web-pages-using-wkhtmltopdf/index.html.
Final solution
So, after some more research we come up with the following command:
[bash]
wget -p -Ppost3 -nH -nd –convert-links http://blogforever.eu/blog/2011/05/17/rendering-and-storing-web-pages-using-wkhtmltopdf/
[/bash]
The two extra parameters are:
- -nd : Do not create a hierarchy of directories when retrieving recursively. With this option turned on, all files will get saved to the current directory, without clobbering (if a name shows up more than once, the filenames will get extensions .n).
- -nH : Disable generation of host-prefixed directories. By default, invoking Wget with -r http://fly.srk.fer.hr/ will create a structure of directories beginning with fly.srk.fer.hr/. This option disables such behavior.
This time, the output files are like this:
[bash]
post3/
post3/9f7efbf7f2902c832da6bd141187265f-bpthumb.jpg
post3/default.css
post3/2401398b6806dd6321b0e97278fd2580-bpthumb.jpg
post3/comment-reply.js?ver=20090102
post3/index.php?ak_action=aktt_js&v=2.4
post3/77e6f2d2239ace354e9b276c0363dffe-bpthumb.jpg
post3/index.php?ak_action=aktt_css&v=2.4
post3/blogforever-logo.png
post3/style.css
post3/l10n.js?ver=20101110
post3/global.js?ver=3.1.2
post3/default.css?ver=2.3.2
post3/robots.txt
post3/4cb1eb4d850a67a41f0303e627e59125-bpthumb.jpg
post3/18a1ba35c3b12649c40d2b747f5f6d1b-bpthumb.jpg
post3/fp7-eu.png
post3/index.html
post3/adminbar.css
post3/jquery.js?ver=1.4.4
post3/815cf9f0cb91ff8810647c9538dec2c2-bpthumb.jpg
post3/stepcarousel.js?ver=2.3.2
[/bash]
Conclusion
Finally, we have come up with a solution capable of retrieving all the elements of a web page and storing them in a single folder. The next step would be to create an archive to preserve these files and store them in a digital repository.