{"id":415,"date":"2014-05-10T14:25:14","date_gmt":"2014-05-10T18:25:14","guid":{"rendered":"http:\/\/jonesling.us\/blog\/?p=415"},"modified":"2015-09-04T15:02:55","modified_gmt":"2015-09-04T19:02:55","slug":"transferring-large-files","status":"publish","type":"post","link":"https:\/\/blog.jonesling.us\/?p=415","title":{"rendered":"Transferring Large Files"},"content":{"rendered":"<p>Linux has an impressive tool set, if you know how to use it.\u00a0 The\u00a0 philosophy of using simple tools that do one job (but do it well) with the ability to chain commands together using pipes creates a powerful system.<\/p>\n<p>Everyone has to transfer large files across the network on occasion.\u00a0 <code>scp<\/code> is an easy choice most of the time, but if you&#8217;re working with small or old machines the CPU will be a bottleneck due to encryption.<\/p>\n<p>There are several alternatives to scp, if you don&#8217;t need encryption.\u00a0 These aren&#8217;t safe on the open internet but should be acceptable on private networks.\u00a0 TFTP and rsync come to mind, but they have their limitations.<\/p>\n<ul>\n<li>tftp is generally limited to 4 gig files<\/li>\n<li>rsync either requires setting up an rsync service, or piping through ssh<\/li>\n<\/ul>\n<p>My new personal favorite is netcat-as-a-server.\u00a0 It&#8217;s a little more complicated to set up than scp or ftp but wins for overall simplicity and speed of transfer.<\/p>\n<p>netcat doesn&#8217;t provide much output, so we&#8217;ll put it together with pv (<em>pipeviewer<\/em>) to tattle on bytes read and written.<\/p>\n<p>First, on the sending machine (the machine with the file), we&#8217;ll set up netcat to listen on port 4200, and pv will give us progress updates:<br \/>\n<code>pv -pet really.big.file | nc -q 1 -l -p 4200<\/code><\/p>\n<ul>\n<li><code>pv -p<\/code> prints a progress bar, <code>-e<\/code> displays ETA, <code>-t<\/code> enables the elapsed time<\/li>\n<li><code>nc -q 1<\/code> quits 1 second after EOF, <code>-l 4200<\/code> listens on port 4200<\/li>\n<\/ul>\n<p>Without the <code>-q<\/code> switch, the sender will have to be killed with control-c or similar.<\/p>\n<p>On the receiver (the machine that wants the file) netcat will read all bytes until the sender disconnects:<br \/>\n<code>nc file.server.net 4200 | pv -b &gt; really.big.file<\/code><\/p>\n<ul>\n<li><code>nc<\/code> will stream all bytes from <em>file.server.net<\/em>, port 4200<\/li>\n<li><code>-b<\/code> turns on the byte counter<\/li>\n<\/ul>\n<p>Once the file is done transferring, both sides will shut down.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Linux has an impressive tool set, if you know how to use it.\u00a0 The\u00a0 philosophy of using simple tools that do one job (but do it well) with the ability to chain commands together using pipes creates a powerful system. Everyone has to transfer large files across the network on occasion.\u00a0 scp is an easy &hellip; <a href=\"https:\/\/blog.jonesling.us\/?p=415\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Transferring Large Files&#8221;<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_crdt_document":"","wprm-recipe-roundup-name":"","wprm-recipe-roundup-description":"","advanced_seo_description":"","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[9],"tags":[194,83,192,191,159,193,195,190],"class_list":["post-415","post","type-post","status-publish","format-standard","hentry","category-linux","tag-big-ass-files","tag-linux","tag-nc","tag-netcat","tag-networking","tag-pv","tag-sam-will-never-do-this","tag-unix"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p4o3FW-6H","jetpack-related-posts":[],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/blog.jonesling.us\/index.php?rest_route=\/wp\/v2\/posts\/415","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.jonesling.us\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.jonesling.us\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.jonesling.us\/index.php?rest_route=\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.jonesling.us\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=415"}],"version-history":[{"count":5,"href":"https:\/\/blog.jonesling.us\/index.php?rest_route=\/wp\/v2\/posts\/415\/revisions"}],"predecessor-version":[{"id":805,"href":"https:\/\/blog.jonesling.us\/index.php?rest_route=\/wp\/v2\/posts\/415\/revisions\/805"}],"wp:attachment":[{"href":"https:\/\/blog.jonesling.us\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=415"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.jonesling.us\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=415"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.jonesling.us\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=415"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}