Hello @fstab @tomwilkie,
I have a working solution to this issue - We had nearly the same case. We have Apache Spark executors running on EMR clusters and we have like 5 executors running per machine. In the current agent's config, you define 1 port and that doesn't work when you have N number of JVMs trying to create a webserver on the same port.

We have an agent, to which you can give range of ports as an config and it will try to start webservers on a port in this range.

Current agent is very simple:

1. read config
2. start server

I had to "mess it up" a bit, in order to accommodate some retries and backoffs. Before I can submit a PR, I need to go trough an internal review process.

My question is, what form and shape should the contribution be? Right now the code changes are in jmx_prometheus_javaagent_common. I have preserved the original code flow when you define 1 port. Would prefer to have the changes https://github.com/prometheus/jmx_exporter/blob/main/jmx_prometheus_javaagent_common/src/main/java/io/prometheus/jmx/JavaAgent.java or perhaps create a different class or even put it to a different module?

Indeed having multiple webservers is a similar issue like i describIed. But I think you still need to know the multiple ports in advance?
My workaround so far has been to write a tremendously simple custom prometheus exporter which is part of our spark worker docker container. It lists and scrapes every jvm process inside the container. Therefore this is not an agent anymore but small standalone application (works with every jvm process)

But I think you still need to know the multiple ports in advance?

Yes you need to know the range you're targeting, have some headroom there and set the rest of the system to that range.

In the spark-submit extra java options the command went from
-javaagent:./jmx_prometheus_javaagent-0.17.2.jar=1000:config.yaml
to
-javaagent:./jmx_prometheus_javaagent-0.17.2.jar=1000-1010:config.yaml

There will be some wasted ports in this solution.

Not sure I agree with the approach. The general approach is to use the Pushgateway and push metrics for these types of use cases (https://prometheus.io/docs/instrumenting/pushing/)

If you are submitting a Spark job where the Java agent HTTP port is dynamic, how are you monitoring it?

With that said... I feel a better implementation would be...

1. Take the contiguous port range and create a Stack of all available ports
2. Shuffle the Stack (Collections.shuffle(stack))
3. Process the Stack, popping the next port value or System.exit if the Stack is empty (port range exhausted)
4. Create an InetSocketAddress with the port
5. Use the HTTPServer.Builder withInetSocketAddress method to create the HTTPServer
6. If there is an Exception, loop back to 3, else continue.

... but again, I don't agree with the approach.

I recall thinking about something a bit more smarter than just going one-by-one from portStart. I like the idea with stack, definitely looks more professional, although I'm not sure if it would improve startup time significantly. When starting up a Spark job with 700-1000 JVMs, it's never instantaneous and it takes a bit of a time for everything to start (speaking from a point when the job is marked as running by YARN). That said, there was no noticeable delay with the JMX Agent.

If you are submitting a Spark job where the Java agent HTTP port is dynamic, how are you monitoring it?

The way it is then scraped will depend on infrastructure layout. In our case, there is no direct communication between EC2 instance and Prometheus. We need to explicitly allow communication for it. It isn't and cannot be truly dynamic as you need to know beforehand which ports to allow in Security Groups / ACLs.
In Prometheus you have a target per port from the range. Let's say we have target_exporter_1000 until target_exporter_1010. Each of those targets will contain a fraction of the metrics, depending on how it played out on the EMR. You cannot really influence what happens on the EMR, you just need to make sure that you won't have more JVMs per node than your defined port range.

I mostly forgot about this issue. Thanks for bringing it up again.
At this point in time my workaround has been to colocate a jvm_exporter with every spark worker process which connects to all visible jvm processes and scrapes jmx metrics from those exporting them in Prometheus format as a single server. This means Prometheus -> jvm_exporter -> jvm_xx (where jvm_xx can be multiple executor processes). This has turned out to be relatively light weight and so far has been functioning without any issues (except the somewhat tricky configuration). Furthermore I would assume this being not bound to spark but could function with any multi jvm setup.

If some one else is having the same issue and willing to implement this workaround I might be able to share the code.

I was wondering if this could be achieved with service location protocol?

4.1. Service: URLs

   Service URL syntax and semantics are defined in  [13].  Any network
   service may be encoded in a Service URL.

   This section provides an introduction to Service URLs and an example
   showing a simple application of them, representing standard network
   services.

   A Service URL may be of the form:

      "service:"<srvtype>"://"<addrspec>

   The Service Type of this service: URL is defined to be the string up
   to (but not including) the final `:'  before <addrspec>, the address
   specification.

   <addrspec> is a hostname (which should be used if possible) or dotted
   decimal notation for a hostname, followed by an optional `:'  and
   port number.

I'm trying to get a good grasp of java's jmx, rmi and slp, but I believe jmx servers could advertise their service URL with port to an RMI server and then from that server jmx-exporter could find those services and their ports and use them.