How to find out if the WAN IP address has changed and get bulk transfers of IP addresses
The system I am developing potentially has a very large number of clients (say, one million) that must periodically update the central server with some information. Clients are written in Java.
A specific use case is for the backend server to have an up-to-date mapping of the IP address to the clients. But client IP addresses are dynamic and subject to (effectively random) changes.
The solution I mean requires clients to ping the server to update their IP address. The period should ideally be once a minute, but even 1 ping / 10 minutes is acceptable.
My questions, in sequence:
-
1 M ping per minute - more than 10 fps. So, first of all, I want to know the approaches can scale to handle such a load. It is to know the available options.
-
Assuming you have more than one solution, which one would be the most cost effective? Cost effectiveness is critical. I don't have my own datacenter or static and fat endpoint on the network, so the server application will need to run on some provider or eventually on the cloud.
Notes:
-
I decided to start the server from home using my own internet connection, but I'm not sure about the performance issues and what my ISP would think about the constant stream of emails.
-
I don't see how the server can automatically detect these IP changes.
source to share
Eric, your problem is much simpler than it sounds.
This problem has been around for a decade, maybe two. There is no need to reinvent the wheel here.
Why polling / ping is a bad idea
Dynamic IP addresses provided by ISPs can have variable lease times, but will often be at least 24-72 hours. Pinging your server every 1-10m would be a terrible waistline of resources, potentially generating over 4,320 useless PER CLIENT HTTP requests in 72 hours. Each request will affect approximately 300 bytes. * 3220 wasted HTTP requests equals 1.3mm wasted multiplied by the target customer account of 1 million clients, you are talking about ~ 1.2TB wasted monthly ! And it's just wasted bandwidth, not any other bandwidth you might need to run your application and provide useful information.
Clients need to be smarter than pinging. Rather, they should be able to check if their IP address matches the DNS on startup, and then send a notification to the server only when the IP address changes. This will reduce your bandwidth and server processing requirements by thousands of times.
What you describe is Dynamic DNS
What you say is "Dynamic DNS" (both a descriptive name for the technology and the name of one company that provides the SaaS solution).
Dynamic DNS is simply a DNS server that allows you to very quickly change the mapping between name and IP address. This is usually useful for devices using an ISP that only provides dynamic IP addresses. Whenever the IP address changes for a router / server on a dynamic IP, it informs the dynamic DNS server about it.
- The de facto standard protocol for dynamic DNS is well documented. Start here: DNS Update API , I think the specific features you are looking for are as follows: DynDNS perform an update . Most of the commercial implementations there are very close to the same protocol due to the fact that the router hardware usually has a built-in DynDNS client that everyone wants to use.
- Most routers (even cheap ones) already have dynamic DNS clients built into them. (You can write your own soft client, but the router is most likely the most efficient place to do this, since your clients will likely be NAT'd with a private IP, you can still do this, but at the cost of increasing bandwidth for public IP discovery)
- A quick Google search for a "java dynamic DNS client" results in source projects like this one: Java DynDNS client (untested, just illustrating the power of lookup)
Other considerations for your system architecture
Let's assume the client IP mapping issue is resolved. You get it and everything works fine, you always know the IP for each client. Then you have a good reliable system for transferring files to clients from mobile devices? I would say no.
Both mobile and home computers can have several types of connections, Wi-Fi, Cellular Data, possibly wired data. Each of these networks can have different security systems. So, connecting a cell phone to a Wi-Fi laptop behind a home router will look very different than a mobile Wi-Fi device connected to a laptop on the same Wi-Fi network.
You may have physical router firewalls that you can fight against. Also, home computers might have Windows firewall turned on, maybe norton internet security, maybe Symantec, maybe AVG, maybe zone alarm, etc. Do you know the firewall issues for all of these potential customers?
source to share
I would suggest it is better to set up your Java program to learn about the IP address change and then only hit the web service.
You can do it,
- when starting a Java program, extract the IP address of the machine and storage it into a global variable or better some properties file.
- Start a batch process / scheduler that will check your IP every 30 seconds / 1 minute for changes. Java Quartz Scheduler is useful for you.
- Calls a web service if the IP address changes.
Thus, it reduces the role of the server and thus the traffic and connections.
source to share
You can create your own protocol over UDP, for example based on XML. Define 3 messages:
- request - the client requests a call from the server
- challenge - the server responds with a challenge (mostly a random number)
- response - client sends username and hashed password + call back to server
Lightweight and not too heavy transport. You can load balance across multiple servers at any tier or using load balancing.
Any average computer can handle millions of such views per minute if you are using the server side in C / C ++ (I don't know about Java network performance)
source to share
Do I have a use case? User community everyone wants to receive photos from each other? Don't want to host images on the server, but share them directly with all users?
There are two questions here. The first question is "How do I know if my own WAN IP has changed."
If you are not NATed then:
InetAddress.getLocalHost()
will tell you your IP address.
If you are NAT then using dynamic DNS and resolving your hostname will work.
The second question is something like "How to exchange photos between hosts that come and go on the Internet."
Possible solution space includes:
IP multicast, possibly with forward error correction and carouseling, for example. FLUTE.
Swapping files - for example. bittorrent.
Post / Subscribe message using Jabber, AMQP, JMS, STOMP or similar. Suitable implementations include RabbitMQ, ActiveMQ, etc. JMS themes are a key concept here.
The solution is to avoid large IP-level execution overhead.
source to share