DNS errors and segfaults after removing shards from a cluster #2073
Description
- OS: Ubuntu 20.04
- Redis: 6.0.5 (AWS Elasticache, cluster mode enabled, TLS encryption and auth enabled)
- PHP: 7.4.3
- phpredis: 5.3.6
I was reducing the number of shards of our AWS Elasticache Redis cluster today in our production environment. After the node group rebalancing the actual node deletion took place and we started getting errors from the application:
Warning (2): RedisCluster::get(): php_network_getaddresses: getaddrinfo failed: Name or service not known
I suspected the enabled slot caching to cause DNS requests to deleted nodes, so I disabled that for a second try on a different cluster.
Indeed this time there were no DNS errors during the process. Afterwards I enabled the cache again and the same errors started immediately. I guess because the old cache was still valid.
So my actual question is: How to invalidate the cache? Where is it actually stored?
A second problem occurring regardless of the cache enabled or not were Apache segmentation faults. I can provide a stacktrace from a coredump, but unfortunately without debug symbols:
Reading symbols from apache2...
(No debugging symbols found in apache2)
[New LWP 299113]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f9a395417d3 in ?? () from /usr/lib/php/20190902/redis.so
(gdb) where
#0 0x00007f9a395417d3 in ?? () from /usr/lib/php/20190902/redis.so
#1 0x00007f9a39541f38 in cluster_send_command () from /usr/lib/php/20190902/redis.so
#2 0x00007f9a3952ca8c in zim_RedisCluster_get () from /usr/lib/php/20190902/redis.so
#3 0x00007f9a3dd0ece5 in execute_ex () from /usr/lib/apache2/modules/libphp7.4.so
#4 0x00007f9a3dd1035b in zend_execute () from /usr/lib/apache2/modules/libphp7.4.so
#5 0x00007f9a3dc8729c in zend_execute_scripts () from /usr/lib/apache2/modules/libphp7.4.so
#6 0x00007f9a3dc26f80 in php_execute_script () from /usr/lib/apache2/modules/libphp7.4.so
#7 0x00007f9a3dd12160 in ?? () from /usr/lib/apache2/modules/libphp7.4.so
#8 0x0000559c9ba78938 in ap_run_handler ()
#9 0x0000559c9ba78ee6 in ap_invoke_handler ()
#10 0x0000559c9ba909ae in ap_internal_redirect ()
#11 0x00007f9a3e683bdc in ?? () from /usr/lib/apache2/modules/mod_rewrite.so
#12 0x0000559c9ba78938 in ap_run_handler ()
#13 0x0000559c9ba78ee6 in ap_invoke_handler ()
#14 0x0000559c9ba91693 in ap_process_async_request ()
#15 0x0000559c9ba91862 in ap_process_request ()
#16 0x0000559c9ba8d926 in ?? ()
#17 0x0000559c9ba82678 in ap_run_process_connection ()
#18 0x00007f9a3e9b2997 in ?? () from /usr/lib/apache2/modules/mod_mpm_prefork.so
#19 0x00007f9a3e9b2c9a in ?? () from /usr/lib/apache2/modules/mod_mpm_prefork.so
#20 0x00007f9a3e9b3a18 in ?? () from /usr/lib/apache2/modules/mod_mpm_prefork.so
#21 0x0000559c9ba5a408 in ap_run_mpm ()
#22 0x0000559c9ba52499 in main ()