Read on Hive.Blog
What is IPFS, Tutorial, and Quick Hosting Guide
Originally published 7/3/2021 on Hive.Blog
What is IPFS?
If you’re familiar with a video I did on the experimental platform
DTube then you are probably at least slightly familiar with IPFS, which
I intend to explain today. IPFS, similar to BitTorrent (which I would
like to point out is legal unless used to pirate content), is a
decentralized file sharing network. Unlike BitTorrent, which focuses on
creating mini networks dedicated to sharing one file, however, IPFS is
one large decentralized network which contains all devices running
IPFS. Similarly, though IPFS shares some similarities in structure to
BitTorrent with a few key differences, the goals of IPFS are also
similar to BitTorrent with a few key differences. The name IPFS is an
abbreviation of “InterPlanetary File System,” and the network is
structured with the idea of building one large network servicing many
clusters of devices.
If you have seen the TV show “The Expanse” you can probably picture the
theoretical use case: if there were networks on a number of planets the
communication within a particular planet could be fast and reliable,
but communication from planet to planet may be slow and expensive. With
this setup, however, a file would only need to enter a particular
planet’s network once before always being available within the network
and no longer be required to be sent from a centralized server on
another planet.
Put in more contemporary terms, consider the idea of inter-continental
communication. Say a file is made available on a server in the
northeast United States. With the normal centralized internet if
somebody in Sidney Australia wanted to access it the data would have to
travel around the world taking extra time and putting stress on every
piece of infrastructure along the way. With IPFS, however, after the
first person in Sydney requested the file the second person in Sydney
would get it from the first, the third would simultaneously get it in
parts from the first and second, and so fourth.
Finally, I’ll give one more example, this one a bit more technical to
better demonstrate IPFS better. Lets say there is three LANs (local
area networks) each consisting of 100 devices with IPFS. Communication
between devices within the same LAN is 200 MB/s. Now, those three LANs
are part of one WAN (wide area network) and communication between
devices that are part of the WAN but part of different LANs is 100
MB/s. Last, any device communicating with the internet (part of neither
the local LAN or the WAN) is 25 MB/s. With IPFS the devices would first
try to download from other local LAN devices, then if that fails they
would try devices on the WAN, and if that still fails they would pull
it from the internet; prioritizing the ideal route first. Additionally,
since the file is then cached on one of the LANs if a device on a
different LAN requests it the file will go over the WAN, not the
internet, saving speed (plus now it’s on two LANs) and so fourth. As
you can probably see, the concept sounds promising, so I will continue
on with the explanation, moving on to how you can use the system
yourself.
Network Basics
Each file on the IPFS network is identified with a hash, which is a
string of text that is a mathematical derivative of the file. The
beauty of hashes is that no matter where the file is coming from the
integrity of the file is guaranteed because any alterations to the file
will alter the hash (alterations to the file’s name will not change the
hash, only alterations to the file’s content). Next, as mentioned
previously, any device that has uploaded or requested the a file will
store and redistribute the file. Each device will continue to host the
file as long as the IPFS service is running until either the operator
chooses to stop hosting the file or the file stops being requested for
some time and is automatically removed from that specific device to
optimize storage. To avoid a file from being automatically removed from
your device you can ‘pin’ the file so that it always remains on it. It
is important that at least a device or two has any particular file
pinned so that it is always available when requested. If you are
attempting to host any of your own content I will delve deeper into
that in the hosting section.
Access & Clients
Now, with the basics out of the way we can finally start utilizing the
network. Starting out here, you have three options: the IPFS client,
web browsers, and gateways. Beginning with gateways, a gateway is a
website which is attached to an IPFS node (for future reference, a node
is simply any device running IPFS) which pulls the file content off of
the IPFS network and provides it to your browser for viewing and/or
downloading. IPFS.io and CloudFlare both have services like this and
allow you to enter in the address followed by the hash to view the file
in the web browser (example: https://ipfs.io/ipfs/-hash-). I am
bringing up gateways first because they are the simplest and easiest
way for you to get access to data on the IPFS network, and great to
check out if this is the first time you are accessing IPFS.
Additionally, they also make it super easy for access on mobile and for
sharing access to others who do not have IPFS installed on their
devices.
As the middle ground between a full fledged client and a gateway, you
can also run IPFS in some browsers. Browsers like Opera and Brave come
preinstalled with IPFS support, and plugins for Firefox and
Chrome/Chromium allow you to manually add IPFS support to those
browsers as well. Using IPFS within your browser allows you to utilize
the network structure benefits of IPFS, but does limit your
capabilities in regards to uploading and/or ‘pinning’ content.
Finally, to fully leverage the capabilities of IPFS you can install the
IPFS client. You can acquire the client at https://ipfs.io and install
it on Windows, Mac, or some Linux distros. With this you will be able
to acquire files based on their hash, host files in a temporary or
permanent manner, and help build up the IPFS network. Furthermore, as
somebody who has spend nearly the last decade experimenting with new
and experimental software, I am very glad to tell you that the
graphical client is super easy to understand – even for my dumb brain.
Since you already know what pinning is, what hashes are, and the basics
of the way the client will behave (such as clearing out files that are
not pinned) you should be able to use IPFS now with very minimal
experimentation. Using the Terminal client shouldn't be an issue if
you're already somewhat familar with Terminal. Also, I should note that
the IPFS client can be used to self host content such as videos and
websites, which I will delve into in the hosting section.
Drawbacks
With this coming to a closing I would like to talk about a few
drawbacks of the IPFS system. One much more minor drawback is the fact
that there are not a ton of people using IPFS. Now, of course with
gateways and the like the average internet user could be accessing
content on IPFS without even knowing what IPFS is, so that is a bit
more of a minor issue.
The other, more notable of the two, is the issue of speed. As I said
before, IPFS is experimental and is lacking full scale adoption. The
biggest tangible effect of this is the speed at which files are
located. Though regularly trafficked files on regularly trafficked
nodes are extremely quick to be discovered, files towards the fringes
of the network are a whole different story. If you place a file on your
device and try to load it’s hash in a gateway it could take upwards of
several minutes for the gateway to actually find the file (I’ve noticed
that CloudFlare can often time out a few times, where the IPFS.io
gateway does not time out but still takes a fair amount of time). Now,
once it has been loaded by the gateway it will be cached for a fair
amount of time so the next time you go to load it (unless it has been
too long) it will be quick, and I will touch on this more in the next
part of this series, but this is worth considering.
Summary
Well, I guess I hope that summarizes everything. I hope you consider
checking out IPFS and playing around with it. It’s certainly a very
promising new system that I expect has a good future ahead of it, and
regardless I find playing around with this type of system to be fun.
Hosting content on IPFS
As I mentioned above, I'd also like to quickly touch on self hosting
content with IPFS. Simply put, the internet is made up of files that
are retrieved from a server and displayed in a web browser, so it
shouldn’t be surprising to hear that a decentralized network meant to
deliver files can be used to host content. For over a year I soley
hosted my website on IPFS, and if you check the links on my main page
you can still find the IPFS mirror.
One of the biggest drawbacks, however, is that with IPFS there can be
no interaction with a server like on the normal web, preventing
services like contact forms, discussion boards, comments, and even easy
edits to the web page. Now, there are some minor benefits to this (no
web analytics, pop ups, or dynamic advertisements without being
creative), but ultimately hosting content on IPFS will limit you to
only a static web page. Though people can be pretty creative within
these limitations, this tutorial will cover the bare bones of such a
setup. From here I can break up this tutorial into three parts:
creating content, hosting content, and assuring access to said content.
Creating Your Site/Content
Since you are likely already aware of how to upload or download content
on the IPFS network, if you are going to use IPFS as a file sharing
service you can simply skip this step as there will need to be little
effort into getting your content ready once you have the file. Even
videos can be uploaded strait onto the network and then downloaded or
streamed by other users. That said, if you intend build a web page
hosted on IPFS then continue on.
As I said before, you are going to be limited in what can actually
implement in a web page on IPFS. Therefore, your web page creating
project might end up looking like it would in the early 2000’s. Because
Nvu was discontinued, I would recommend installing the HTML editor
KompoZer to create a word document like site which contains everything
within a single page (I told you this was going to be bare bones,
though it can have multiple pages as well). Proofread it well, because
it will be a pain to make changes, and get your HTML file ready.
Alternatively, you could export a word document as HTML using MS Word
or LibreOffice, though software dedicated to HTML (either KompoZer or
your editor of choice) will probably be better. Finally, it would also
be possible to just use a text document instead of an HTML document,
but that would be really limiting your creative expression options, so
again, I would just go with HTML.
Hosting
If you want to have a website available, then you will need to know
where it will reside (as it would be pointless to make a site that ends
up becoming inaccessible). Your first option would be to try to host it
yourself. This of course would require a device that is always online,
so it’s certainly not for everyone, but it is worth considering. Simply
pin your file and leave the device or VPS running and DHT will work
it's magic (no static IP required).
Secondly, it may be possible to rely on the network itself to host your
content. If it is accessed enough then it would be hosted by all the
individual nodes accessing the content – especially if it is content
that you expect others to pin specifically to assure that the content
remains hosted. That said, for a lot of us we can’t assume that others
will host it.
Third, you can always play a game of whack-a-mole with public gateways
and the like. A week or so before writing this I used the free version
of global upload (which will temporarily host a file but requires
payment to actually pin it) to upload a text file that I then loaded in
several gateways. I let it sit for a week or so and then checked it and
it was still there, so if a file is small enough and you go out of your
way to request it on a bunch of different gateways regularly it could
probably just float around in the network. Furthermore, if it does get
discarded on all the nodes at some point you can re-upload the file and
request it to get things going again. You also may want to couple this
along side self hosting if you were already planning on self hosting
that way you can be sure that your files are quick to access.
Finally, there are also paid services that will host your content for
you. While this may seem like we just went full circle back to regular
paid internet hosts, for most people who would want to host content on
IPFS this is probably the best. As I mentioned global upload in the
previous option, they currently have paid options that would allow you
to permanently pin content. Additionally, services like Pinata have
both free and paid pinning services. Pinata’s free service allows 1GB
of bandwidth, and their next plan up starts to charge you $0.15 per GB
of bandwidth that you use. Not only are these very fair prices, but if
you are hosting a simple web page than these are amazing! If a simple
HTML website was 10 KB that would allow it to be accessed 10,000 times
a month on Pinata, and furthermore since if it was accessed 10,000
times a month it would likely be distributed among the network instead
of solely being on Pinata - meaning you would get a whole lot of
addition access as well. I personally have used Pinata for a while to
host my IPFS site, and never even used remotely close to the 1GB limit
of the free plan.
Sharing Content
With content created and uploaded, the final thing is to make sure your
content is easily accessible. You should have your hash that can direct
users to the file, however, ideally you want to have a clickable link
that will work for users who may or may not know how to use IPFS. Also,
some of these options may allow you to change the hash it directs to
and thus modify content on your website. It is worth noting that IPFS
does has a service called IPNS, which is essentially a hash that you
can use to point to any hash of your choosing. This would be perfect
for easily updating web content, however, because it is a step or two
outside of the scope of this basic tutorial – and because IPFS’s speed
may make IPNS very slow – I will be leaving it out of this guide.
Anyway, I’ll cover three different methods of doing this below:
First, aside from sharing your hash on it’s own, the easiest way to
have a clickable link that requires no understanding of IPFS is to
simply share the link to a gateway that is open to your file. The only
drawbacks are that the link is a bit long and you can’t change the hash
without changing the link that you shared.
Example: https://gateway.pinata.cloud/ipfs/QmW7S5HRLkP4XtPNyT1vQSjP3eRdtZaVtF6FAPvUfduMjA
Secondly, if you want to share your content in standard web form you
can use shortened URLs or redirect pages to get the user from a link to
a gateway with your file’s hash on it. Services like Bit.ly allow you
to shorten the link to a gateway and share the link with others.
Similarly, if you already have a normal website you can create a
redirect page (say example.com/ipfs.html), which when loaded will
immediately redirect the user to the gateway page that is loading your
file on IPFS. In either of these cases you can also generally change
the link to direct people to a new IPFS hash if you need to change your
page or content.
Example: https://nathankmbowie.com/ipfs.html
Finally, you can always use a domain, a subdomain service, or a
subdomain of an existing domain that you own to direct a user to your
IPFS website or content. In any of those cases you can set the domain
to either redirect the user to an IPFS gateway service or use something
like Cloud Flare's IPFS services to configure the domain to connect to
IPFS content like it would on the normal web.
Example https://ipfs.nathankmbowie.com
Hosting Summary
Well, I hope you found this interesting, useful, or both. I think IPFS
is worth taking the time to understand and experiment with, and you
might just find it fun to toy around with as well.
All content in /tutorials is
licensed Creative Commons
Attribution-Share Alike
unless otherwise specified. Feel free to share it around and re-use it,
but please give me credit if you do so (and would love if you sent me a
message or something). You can also
send me a tip if you found this helpful, but no
pressure - this is a hobby.
Contact Me:
Mastodon @natebowie@mastodon.social
Twitter @NathanBowie6