Idea

In the last month the main topic of any news site, in my country, are forest fires and how they are destroying everything in their path. They are causing major issues burning down houses and large fields and owners are getting very little back or nothing at all. Everyones excuse was that they didn't have time to prepare and that when they got the call it was already too late. We had to change that so we tryed to find a way to help but that it doesn't cost so that there aren't any excuses why it shouldn't be implemented.

We had a weather station design that sends temperature, humidity and pressure data to our server using ESP devices and our existing mesh network, it was inexpensive and low-maintenance so we used it as a base for our fire detection device. To detect fires we added an air quality sensor that could detect smoke but also combined with other sensors like the temperature one it is possible to detect if that is just smoke or if it was already a large fire because of the temperature difference.

The next step was figuring out how to set up these devices over a large area and also how would they send their data without an internet connection. We needed to find a way to transfer data over distances greater than 2 kilometers and it needed to run on batteryes for a long time, the solution was the use of LoRa.

Simple schematic of how the whole fire alert system would work

What is LoRa?

LoRa Technology is a wireless modulation for long-range, low-power, low-data-rate applications. By achieving a range of more than 15 kilometers in a suburban environment and more than 2 kilometers in a dense urban environment, LoRa technology solutions target multiple application domains, such as Internet-of-Things (IoT), metering, security, and machine-to-machine (M2M).

This basically means that we can make devices that could run on batteries for weeks and also be able to send us data over 10 or more kilometers because it would be used outdoors and that is enough to cover a large area.

Dragino shield for Arduino Uno to use as a LoRa node

Testing

To test our devices we needed to test how far we could go and still be able to send data to our gateway using LoRa and how long can it run on a battery pack. For measuring the distance we activated our gateway and sat in a car with the LoRa node that sent data every 2 minutes to our server. We recorded our route and marked whenever we received the signal. The test was done in a city and the furthest we got was 2 kilometers and that was pretty amazing for sending data in a crowded city with many obstacles. For the battery test we plugged in a battery pack and the device was set to send data every 1 minute and when it got depleted we measured the time and then we set it to 2 minutes and then to 3 so we could predict how much it would last with other options and to find the best one.

Delivery

The delivery of our nodes and how the system works is explained in another log: (link) For delivering nodes we are using drones with release systems attached to them that drop the nodes when they fly over a set place. This makes it easier to place a lot of nodes quickly but all from a single location.

What is next?

We still need to make an enclosure for the LoRa node that is resilient to outside effects and different impacts. After that we can test the delivery system and also how far can we place the LoRa nodes and how long can they last on batteries outside.

From Jun 7th till Jun 11th Pannonian Challenge XVIII-2017 was held, one of the few festivals in the world dedicated to extreme sports such as BMX, skate, inline and MTB competitions. There are also numerous concerts during the festival which makes a perfect combination of sports and music. Festival attracts many lovers of extreme sports, athletes, citizens od Osijek, visitors from other European and world countries, both young and old. 

MeshPoint team was proud to provide the whole network setup for the entire event – from public Wi-Fi (otvorenamreza.org) for all attendees and private Wi-Fi for Management, sport judges, press corner, staff  and for point of sale. 

We have been partners with Pannonian challenge for last 6 years, and this is where we have tested and improved our mesh technology. Each year thousands of people come to this event because of that we had opportunity to test our technology in real world conditions, and MeshPoint has been proven to work with hundreds of clients connected at the same time. Thus when we decided to act during humanitarian crisis we were sure that our technology can work in most demanding conditions.

I will go through every step in the process of planning and realization. Let’s start with obstacles in planning the network. The skate park is located on the left side of river Drava in Osijek right by SRC Copacabana. The network infrastructure there was broken and in disrepair so the first and foremost thing for us to do was bringing high speed Internet to the location.

                                                      space that we had to cover with Wi-Fi

We used Ubiquiti LiteBeam AC 23 for point to point links from the location of one of our backhaul nodes where we have 200/100Mbps fiber optics connection. From backhaul location there was no line of sight to where the event took place. Thus, after some link planning we made two point to point links – the first leading from our backhaul site to one high apartment building and the second from that high apartment building to the skate park. That was our primary link. For backup link we used another 80/80Mbps link that was provided by event sponsor - Vipnet telecom.

All backhaul links were installed couple of weeks before the event so we had time to test the reliability of both links. May I add I was very satisfied with the results for both links were rock stable and operating at their full capacities. When the backhaul networks were set and tested we started first of probably 1000+ iterations of device physical and logical plans (placement and address and bandwidth planning). That is one off the best parts of Meshpoint, i.e. you really don’t have to do anything special yourself - set it, (give it power and Internet) and forget it.

             Wireless link from our backhaul location to the apartment building (all specs are correct )

                                     Wireless link from the apartment building to the event location

                       Backhaul links (top: Vipnet 80/80Mbits, bottom: Otvorena mreza 200/100Mbits)

For public network we deployed 5 Meshpoints that were strategically placed in the field and connected via 2 Mikrotik RB260GSP POE switches to EdgeRouter X SFP router flashed with our open source firmware which is using our primary link to the internet (200/100). Router was used for monitoring the traffic, throughput and number of clients.

                    Red dots indicate final Meshpoint positions after the stage of the event was set up

                                        Our volunteers promoting Meshpoint at the event

As for the private network, we used a couple of Unifi AC Mesh devices and 2 Unifi AP AC Lite (also monitored by our Unifi Controller server) connected to our private core switch Mikrotik CRS112-8G-4S-IN which was trunked to Mikrotik 3011-UiAS-RM (load balancing + failover and main DHCP server for the network) and to Vipnet provided Cisco 2901 Router.

                                      Network setup after all the changes and additions :) 

Now that you know the equipment and specs of the network, let’s talk statistics by day and the end result.

Day 1. we were all nervous and anxious to see the network perform. That first day we were stuck to our screens monitoring performance and hoping that it works both because of the event itself and because this was our biggest deployment of MeshPoints. By the end of the day, ones the results were in, we saw that network was stable, there wasn’t even a hiccup and MeshPoints did their job perfectly. Most number of concurrent connections were around 230, with a capacity for more. By the end of the first day we saw around 6000 unique clients.

As you can imagine day two went exactly the same. Backhaul stable, local wired network stable, Wi-Fi network stable, clients satisfied and a lot of pictures/videos uploaded to Facebook, Instagram, and other parts of network and social media. The trend of rock solid Wi-Fi network at the event continued and by the end, i.e. at the peak there were more than 500 people connected and using the network at the same time. Overall, over 6000 clients used our services in a 5 day period.

                                                                          Final day statistics

Of course it wasn’t all roses, we had some problems but it was mostly during the week before when we were physically setting up the network. 90% off all problems had to do with broken cat5e cables. Also, at one time when I freaked out that nothing was working and later found out it was our local power company putting stronger power lines for the event. Needless to say that we got an UPS in the NOC and after that it was all smooth sailing.

                                    Fixing some old broken cables and putting new lines

This was also a great opportunity to develop our first dashboard so we could track all the statistics necessary to monitor and maintain the network during the festival, but also to further plan and 'fine-tune' our services for big outdoor events.

All of our pictures from Pannonian Challenge 18 you can find on following links:

https://photos.app.goo.gl/0Yxw2n6eLUCKqvDj1

https://photos.app.goo.gl/gTmtvbG1I0RT7oeg2

I must say a big thank you to all the volunteers at the event and the whole organizational team and my colleagues form Otvorena mreza for making the whole thing possible. We can’t wait till next year to do all of this all over again but Bigger, Better, Stronger!!!!!!!!!!!!

IoVuT - Internet of very useful Things

In London at Europas conference we received award for best humanitarian innovation, this was a huge things for us, because this award was given to us directly by Unicef.

Just few days after the conference ended we've recieved a call from Unicef to establish an IoT network in Zaatari, Jordan, one of the biggest Refugee camps, with population of over 80,000 people. mostly Syrian civil war refugees.

Zaatari camp was opened in 2012, at the beginning of Syrian War. Since the beginning, the humanitarian organizations have been struggling to keep up with growing demands of the camp's inhabitants.

Uber for waste (IoT for good bad things?!?)

On the call Unicef team from Jordan and Lebanon explained why they needed our help and why they gave us this award. We got this award because they said the bigges issue they face in the field today is setting up communication, and then recognised MeshPoint as an great solution to this problem.

But project they wanted our immediate help actually had nothing to do with providing Internet, but with something completely else - human waste.

One of the biggest issues currently in most refugee camps is managing septic tanks, and emptying them in time. Because if this is not taken care in proper time then with overflowing sewage diseases spread, and most often first to be affected are the most vulnerable - small children.

Right now, we can help, by providing a backbone for a network of IoT sensors that would collect levels from clean and dirty water tanks. First step will likely be installation of water level sensors in water reservoirs and waste disposal tanks. The sensors will be connected via Meshpoint network to a server with an SMS gateway, alerting the companies responsible for the logistics when the water or waste reach determined levels, speeding up the process of delivery, cleanup, and cutting down costs and preventing hygiene catastrophes. The volunteers are already calling it 'Uber for waste'.

Clean water and waste level sensors + MeshPoint = much less diseases.

While we negotiate our involvement in this project, we're always looking forward to cooperate on similar projects in the future.

Future plans

In our IoT networks we predominately use ESP8266 chip as a basis for our sensors. It's a really powerful chip with integrated WiFi and a very economic sleep mode, that can last over three weeks on only one (used) Li-Ion battery, provided it 'wakes up' every five minutes. (We recycle batteries from old laptops and similar devices).

When ESP wakes up, it searches for WiFi, connects and sends data to a server and shuts down. We have measured the time it needs to do all this tasks to 7 seconds. The ESP asks for DHCP from the router, and a few more messages are exchanged, which is time consuming. Some time may be saved by implementing a static IP address. We'll starting our measuring experiments soon. We expect with 4 cells to get over a year of completely autonomous sensor collection. Here is how our battery measurements are tracked:

In our DB servers, we have implemented triggers that send out an alert roughly one day before the battery is totally depleted and damage can be done to it, so we're always able to change it before the critical moment.

On our roadmap are also air quality sensor stations, which can be put in forests, and in combination with Meshpoint can act as a early detection mechanism for forest fires and also a climate monitor and analyzer tool

Wifi mesh towers

To provide internet to stationary humanitarian teams who had tents and usually solar panels or generator for power we started building wifi towers. Wifi towers consisted of these parts:
- camera or outdoor speaker tripod
- outdoor waterproof box or case
- Ubiquiti Toughswitch (5 port POE switch)
- Dovado TinyAC router with usb port (router used for setting up LTE link)
- Huawei E3372h LTE USB modem (which can be setup to use external LTE antenna)
- 3x Ubiquiti Loco M2 wifi routers (for hotspots)

This setup worked great, and we created quite a few of them. We build them for Greenpeace, International Organisation for Migration, for Jesuit Rescue Service and for quite a few smaller NGOs.

Have you tried turning it off and on again?

This setup worked great... until it didn't. We started getting calls and getting out in the field and the issues was that because things still were quite chaotic humanitarain NGOs were moving their tents to different location almost every few days...

And when they needed to move they had to take apart our wifi mesh tower and put it back together again on new location.

This is really easy to anyone with at least basic IT knowledge, but this is not what first responders from humanitarian organisations were trained to do. Whey were great at their job, but this wasn't part of it, and they relied heavily on having stable connectivity.

We would come each time and help them with what usually was a cable which was plugged in wrong port.

Eureka moment

By working with the first responders from all the biggest international humanitarian NGOs we saw that there is no device suited for their need. A device for humanitarian workers must me smart and powerful but first and foremost - easy to use. This is where idea for MeshPoint was born.

First prototype

First prototype were three wifi routers in a big water pipe with just few 3D printed parts.

Second prototype

Then we created second prototype with all 3D printed parts.

This prototype worked great as proof of concept but has issues, mostly because radios weren't isolated wifi boards caused created interference for each other. Other issue was that all routers used omni directional antennas so they send wifi signal in all directions, but they also picked up noise from all directions.

All these issues made this made issue for this prototype so it had shorter range then expected and when more than 100 people connected it would just stop working all together.

But we got lots of valuable lessons from this prototype and moved further.

Third time is the charm

With third prototype we used all we learned so was and build a device which was much better. It used three separate sector antennas and has shielded radios. Also because it was POE powered and it had POE in and POE out we could daisy chain them and make assembly much, much easier.

How come there is no power socket in the middle of the field?

After we solved getting internet uplink connection working we needed to tackle one other big issue - power.

We take power (and most things) for granted, but we found our the hard way that during crisis events you can't rely on having power. And sometimes you have to provide Internet in the middle of the field, and there are no power sockets in the middle of the field.

So even when we got Internet connection up and we could get out mobile hotspots to share the Internet we would run out of batteries after few hours and would need to swap them out.

So we were thinking on using solar power but because we were in the field all the time we got in person with minister of Internal affairs and he allowed us to setup hotspots inside refugee camp.

So we were looking how and where to install out devices, and first location that looked promising were light poles.

Light poles were great location because by installing 12 hotspots on 4 towers we could cover whole refugee camp.

But there was one big drawback, light poles had power only during the night, so during the day when most people would like to gave Internet connectivity we wouldn't have power.

Of course we knew that devices such as UPS exist, but we decided UPS are not a good solution because of these reasons:
- we couldn't afford ones we needed (ones with 4 batteries, which are really expensive)
- we couldn't source high capacity UPS locally or in short time
- UPS converts DC to AC and this conversion has up to 40% loss of energy
- UPS aren't build to be used outdoor

So we started to build 24V (most outdoor wifi devices use 24V POE power supply) DIY power solution from this parts:
- 24V car battery charger
- 4x 7Ah 12V batteries (ones used for UPS and scooters)
- Arduino based voltage and temperature monitor connected to OpenWrt router via serial connection
- big plastic box
- 8cm thick sheets of styrofoam for isolation

Batteries were connected two in series and two pairs in parallel. We didn't have time (nor money) to add or build proper over and under-voltage protection for batteries, but what we did was to carefully balance (with additional calculation and measurement) that batteries will hold this load and that charger will properly charge but not over charge batteries and that wifi devices won't over use and under voltage batteries.

This system worked for all 8 months that refugee camp was operational. We have only one tower fail after few months, but others worked without any issue all this time.

Hackerspace heroes to the rescue

I co-founded #labOS hackerspace in my home town and when saw that there is no way I can build all that is needed in 3 days I knew that only was I was going to make it is with lots of additional help.

When I asked people from out hackerspace to help out me response was overwhelming.

Without such great support there is no way we could have made this happen!

We managed to get four complete kits build, tested, connected to see if Arduino was collecting and sending data, and once everything was done disassembled it and prepared it for transport.

Hope you have life insurance...

In order to deploy wifi hotspots and diy power system we needed to climp 20m high towers that were very rusty because they haven't been used or maintained for over 20 years.

In order to climb these towers I hired professional climber who trained me before to climb building, but this was something else, and much harder.

Once we climbed up we were very happy :) But then we remembered that we also need to bring up all that equipment we prepared...

So to bring up all devices, connect them and install them on all four light poles it took us 6 hours of non stop work. We were totally exhausted after that, but also very proud of what we acieved.

How it all started

In August of 2015 whole Europe was part of unprecedented humanitarian crisis in which hundreds of thousand Syrian refugees were fleeing from their war stricken country in hope to save their lives and for a better future.

Overwhelming numbers of Syrian refugees that wanted to enter the EU surprised all humanitarian NGOs and caught them off guard.

What we saw in the news and at the border crossing was total chaos...

When humanitarian or natural disasters occur the need to communicate is immediate. We have witnessed first hand how the children who are most vulnerable are affected because rescue teams are not prepared to tackle this issue. Current solutions are too slow, require highly trained people to deploy it, don't scale and are very expensive.

Because mobile networks were not designed to handle such big number of people (refugees, teams from multiple humanitarian organisations, police and volunteers) in small area the whole system collapsed and you couldn't call anyone or get data connection up to the Internet.

First try and first fail

Out first try was with few routers in backpacks with batteries and 3G modems. This approach mostly didn't work. But what we was in the field was just how important it was for refugees to get any kind of connectivity so that they could use their smart phones to let their families and friend know they are aright.

Survey conducted by International Rescue Comity showed that first question that refugees asked when they landed on Greek islands was "Where am I?", second question was "Do you have Wifi?" and only the third question was "Where do I get food and water?".

Communication taking precedence over food and water showed us just how important it was for refugees to stay in contact with their families and friends.

Out approach mostly failed, because we were trying to connect via 3G modem towers that were not designed to handle this much people trying to use the network at the same time.

Silver lining

What we saw in the field also shocked us, it wasn't only us who were trying and failing to setup Internet connection during this humanitarian crisis, but also all the teams from the biggest humanitarial NGOs were also failing. We saw Greenpeace, Red Cross, Unicef and other teams from smaller organisations trying to setup any kind of communication with their HQs but they didn't succeed what ever they tried.

This is what motivated us to try even harder and try different approaches and different pieces of equipment.

We found out that 4G network was not overcrowded so after getting LTE 4G modems we managed to get uplink to the internet.

After that we also managed to connect to 3G towers but we used long range antennas to connect to remote towers instead of local ones which were overcrowded and not working.

Now we had Internet but all most wifi routers we tried couldn't handle more that 20-30 people at once. One NGO team is around 20 people, and also there were few thousands of destitute people, children and whole families who asked us for Internet access...

So first part of the puzzle fit in place, but this was only first piece. We knew we were on the right path, but there was still lot of work to be done.

While providing Internet network for one of the East European's greatest international extreme sports competitions, Pannonian Challenge XVIII-2017, we developed a dashboard in order to better monitor:

• current number of wifi clients
• currently connected wifi clients on all MeshPoints
• wifi clients distribution per MeshPoint
• total number of clients
• total number of clients on MeshPoint hotspots
• download and upload traffic through time
• ultimate download and upload traffic
• data usage

In time, we will see whether it needs some improving or changes depending on type of hotspot/network employment (crisis situation, social event, or other).

What is sure is that it will enable easier monitoring of hotspot/network performance indicators and ease the writing of different reports. :-)

While in Kutina we upgraded MeshPoint components with ones that have more internal flash memory for additional software packages for remote management. We also did a speed test and network range check.

Because of the architecture of Kutina assylum center and the layout of the buidling itself, there is poor network quality in some parts of the center. So we made plans with the center officials to expand the current wifi network by placing additional MeshPoint routers on different locations. We took photos of the place and talked to center beneficiaries to learn more about their needs and to better plan the position of new MeshPoint routers and further coverage of the network.

In Zagreb we updated the firmware (from OpenWrt to LEDE) on the existing MeshPoint router. We checked the network range and we did a speed test on central MeshPoint used in the main lounge area. While talking to center beneficiaries we found out that network quality is great and if there are any issues, they are almost always connected to network congestion caused mostly by huge file downloads and video streaming.

This winter (December, 2016), as a part of local wifi network project conuducted in collaboration with the city of Osijek, Coratia we installed a number of MeshPoint devices in one city block, which gave us an opportunity to test MeshPoint's endurance in low temperatures and extreme weather conditions such as snow, cold winds and freezing rain. It is now officially confirmed that MeshPoint is truly weatherproof and that it can stand low temperatures down to -22°C without any problems.

It is very important for us to stress test our equipment because we can never know how many people are going to connect to our network and we need to know how to prepare for every situation.

For that reason we are logging every data necessary to run our mesh network at full strength and find any weaknesses that could arise and cause the network to slow down or break. That is a big problem because in crisis situations it is very important that everything works as planned.

For logging our data we use custom made scripts that record the number of users, data usage and bandwidth and send all of that data to a server running InfluxDB (used for recording time sensitive data) and Grafana for the representation of the data. We also have a dedicated page that is much simpler and accessible to everyone else to see the data in real time.

In the pictures included you can see some of the data that we gather and a screenshot of our page for representing the data.