community 
The multiprocessing doesn't seem to work for me
I have a healthcheck function that I use for multiple resources. I am trying to run this healthcheck function parallel for different resources in a blueprint. Here is part of my code :
for command in res_commands:
pool = ThreadPool(len(resources))
if 'healthcheck' == command.Name:
try:
if value == 'ViasatSmac':
globalInputs = helpers.get_reservation_context_details().parameters.global_inputs
vasn_val = globalInputs['vasn_ip']
input_name_val_command = InputNameValue('vasn_ip',vasn_val)
api.ExecuteCommand(self.reservation_id, root_resource_name, 'Resource', 'healthcheck',commandInputs=[input_name_val_command])
else:
# api.ExecuteCommand(self.reservation_id, root_resource_name, 'Resource', 'healthcheck')
async_results = pool.apply_async(self._async_healthcheck, api,root_resource_name)
except Exception as e:
api.WriteMessageToReservationOutput(self.reservation_id,e.message)
pool.close()
pool.join()
res = async_results.get()
api.WriteMessageToReservationOutput(self.reservation_id,res)
.
but when I ran this code I got the below error:
TypeError: _async_healthcheck() argument after ** must be a mapping, not str
I am new to multiprocessing and I am not sure what's wrong here?
and here is the function that I am using in apply_async:
def _async_healthcheck(self,api,root_resource_name):
reservation_id = self.reservation_id
api.ExecuteCommand(reservation_id, root_resource_name, 'Resource', 'healthcheck')
Hi @gjavadi
1. don't create the pool inside the loop - otherwise, you create a new one in each iteration. so create the pool outside of the loop, then after the loop use the pool.close/join
2. the apply_async gets the function name (_async_healthcheck) and then all of the arguments together as one argument inside parenthesizes. So in your case, it should look like this:
async_results = pool.apply_async(self._async_healtcheck, args=(api, root_resource_name))
3. the results that you get are also a list of objects to you need to iterate on them as well:
for result in async_results:
res = result.get()
print res
4. make sure you also handle the other execute command (line 9 above), otherwise it won't run in parallel to the others (or maybe just in parallel to some of them, and the others will wait for it to complete).
Yaniv
Hi Yaniv, Thank you so much for your help. I will try all of those thing and I will let you know.
Yes, sorry, in your case, the result that you get is not a list, so depends on where you'd like to check/print it.
so one way would be to store the results in a list:
async_results = [None] * len(resources) idx = 0 ... async_results[idx] = pool.apply_async(self._async_healtcheck, args=(api, root_resource_name)) idx += 1
then after the pool.join you can iterate as before.
The other option, if you just need to print the results, is to call a callback function when a function ends:
def print_result(result):
print result
and then in your loop call the async like this:
pool.apply_async(self._async_healtcheck, args=(api, root_resource_name), callback=print_result)
Yaniv
Thank you Yaniv, the result that is an argument to the print_result is not something that I passed? it returns automatically the result from calling my function(_async_healthcheck)?
Hi Yaniv,
I have modified my code based on your comments and thanks to you it worked, but at the end I am getting another error that I am not sure:
\sandbox_scripts\environment\setup\setup_script.py", line 289, in _call_healthcheck TypeError: 'ApplyResult' object is not iterable
for result in async_results:
res = result.get()
api.WriteMessageToReservationOutput(self.reservation_id,res)
and the other problem that I have with this, I have 4 resources that i am trying to run the healthcheck in parallel but it starts with three of them and for the fourth one it hung for a while and at the end it throws the top error?
@Yaniv.K I am so sorry that I ask so many question but when i have three modems in the same blueprint and run the healthcheck in parallel , it always hung on the last one and because i have these healthcheck on my setup script , the setup script can't finish the job? Am I doing something wrong here? I tried without adding this multiprocessing code and it can finish setup script.
Hi @gjavadi
Not sure what might cause the last one not to run/fail without seeing the whole script and knowing more about your environment (resources/drivers/how many commands slots you have/etc).
You can debug your setup script from the IDE instead of running it from CloudShell. For that, you'll need to "attach to cloudshell" in your script, read more about it here:
https://qualisystems.github.io/devguide/orchestration/8.0.0/scripts-deep-dive.html (look for "debugging scripts")
Let me know if you still have issues, or update us when you find what was the problem.
Yaniv
Hi,
I did that but unfortunately the same thing happened in my main and I never get back the result. I know you are busy but if you have few minutes, can we have a quick session?
