Slow moving hosted database

Options
I know there is a knowledgebase solution for this, but I wanted to hear from hosted users. What is the first action you take when the database starts to move at a snail's pace? Do you have any tricks worked to get it moving at a normal speed again?

e7ce04ee1c5fcf2feb16479757bc3753-huge-fu


 

Comments

  • Sunshine Watson:

    I know there is a knowledgebase solution for this, but I wanted to hear from hosted users. What is the first action you take when the database starts to move at a snail's pace? Do you have any tricks worked to get it moving at a normal speed again?

    e7ce04ee1c5fcf2feb16479757bc3753-huge-fu


     

    Probably not very useful, but as a long time user of Windows, I usually reboot the browser and if that does not help, I reboot my computer.


    Jeff

  • Sunshine Reinken Watson
    Sunshine Reinken Watson Community All-Star
    Ancient Membership 1,000 Likes 500 Comments Photogenic

    Jeffrey Lawson:

    Probably not very useful, but as a long time user of Windows, I usually reboot the browser and if that does not help, I reboot my computer.


    Jeff

     

    Actually, that is helpful! Rebooting my computer is what worked for me today, but that isn't always the case. It's funny that no matter how much technology advances, turning it off and on again still usually fixes issues.

  • Jennifer Lucarelli:

    sometimes disconnecting from the Citrix client server through the connection center helps - but then when you reconnect, just try to make sure you are on a different "server number". You can view this in the Citrix Connection Center. you may have to disconnect multiple times before it switches you to a different server number.

    Jennifer, could you give more specific details about this?  I know that there are times when just a few of our users are slow and this would help with troubleshooting when this occurs.  Thanks, Annette

  • John DeMarce:

     

    I reported it to BB and was asked to do a bunch of trace reports. Who has time?? The recent slowness is not just a problem for me but is a problem for every user in our office. Based on recent performance, I would have to reconnect or reboot about 6 times a day. That is not a proper solution to the issue!!


     

     

    I've contacted support about intermittent slowness recently and been told that the episodes were due to known issues on the data center end and that they're being worked on.  No notice of the degraded service was given on the hosting status page though (or Blackbaud Support Twitter or any other source), and no details were available about the nature of the problems or when solutions are expected.


    John,  we've experienced times where all of our users are experiencing slowness and the issue is the data center or our database rather than the individual connection to our hosted environment.  Fortunately most of our slowness issues occur with a few users and logging off and waiting a while to reconnect via Citrix resolves it usually.  I was told that it's important to report when you experience slowness for all of your users to Blackbaud Support .  I know that this is very time consuming to troubleshoot but it would pay off if are able to determine the cause and fix it if it occurs on an ongoing basis.   Best of luck with successfully resolving your hosted environment performance issue(s).   Annette

     

     

  • hi Annette. not sure what other info I could provide.... once logged in, you can usually open the Connection Center from the bottom right corner of your screen. Right click the citrix icon, and then select Connection center. pic attached. Select the server, and then "disconnect" or log off. Then, when you log back on, return to this spot to see if you connected to a different numerical server. This should help troubleshoot.
  • Joan Perry
    Joan Perry Community All-Star
    Tenth Anniversary 100 Likes 100 Comments Photogenic
    Thanks for meme and the laugh Sunshine.


    Have to say I am in the realm of logging out and back in, then rebooting the computer and then checking with other users AND the hosting status page. I ALWAYS check the hosting page before contacting support...not that it means that page is truly current and/or accurate.

    Have to remember to add the checklist item about the Citrix Connection Server and checking the server ID (Thank Jennifer L!).

    Plus checking access to other online sites, trying to determine if it's our internet access or something else. Though if you've done any testing of connecting with Citrix, we make anywhere from 10-20 stops between ourselves and Boston.

    As time consuming as it can sometimes be to reach out to support, when it is problematic for a number of users in our office, I do reach out and share the above. Sadly, the comments about Hosting and the Status page are sometimes discouraging..."nothing has been reported"...I try to keep on my patience hat and let them know that is why "I am reporting problems." Reporting issues is our best offense.
  • Sunshine Watson:

    I know there is a knowledgebase solution for this, but I wanted to hear from hosted users. What is the first action you take when the database starts to move at a snail's pace? Do you have any tricks worked to get it moving at a normal speed again?

    e7ce04ee1c5fcf2feb16479757bc3753-huge-fu


     

    Hi all - I had severe issues this morning (waiting 3-5 minutes between each click) and got on the phone with BB. Funny thing is that by the time they answered, it has "fixed itself". Anyhow, the guy gave me this Knowledgebase article: https://kb.blackbaud.com/articles/Article/42699. Check it out...

  • I also have an open case regarding slowness in the hosted environment.  All of my users have reported a very sluggish environment this past week.  All are working at different campuses with different internet connections and in different applications.  I will admit that the routine for hosted support is to first check different machines, different user names, different internet connections, then ping their data center, then do a 24-hour plot.  I'm also finding that the problem does not appear to be on our end.
  • Thanks for the note. I really appreciate it. It is my belief it appears to be a BB issue, especially if you pay attention to the community posts like ours. 
  • Aaron Rothberg:
    I've just opened another case (13507050) for latencies this morning. I made it clear that the latencies have persisted for months, that all database view users are affected, and that all other network applications in our office do not experience latencies. Here are a few of the responses I got back from support.

    "The benefit of moving to NXT webview is that you have the cloud and everything runs as fast as your internet connection"

    "Unfortunately, with hosting, there can be latencies based on the number of people logged in at the time. We do our best to prevent any delay. "

    "there haven't been any issues listedfor this week on the Boston servor"

    "If needd, you can send a pingplotter to us using step 5 to see if there is a congestion somewhere that we can rectify."


    I'm not able to run Pingplotter for 24 hours without locking my computer (is there anyone still out there that does not have an IT policy that auto-locks your computer after a period of inactivity?), but I don't think I need the 24 hours of data. The latencies are clearly happening at the hop just before I connnect to the data center.


    329ad2b5cb1cf8495185854cff29d641-huge-sc

    7a9eabefd4850c14ef4b4b9328e5c999-huge-sc

    0ebce4063c0ea2e26a19930eb1b623eb-huge-sc



     

    Getting very similar results:

    PS C:\\Users\\demarcej> tracert ctx-login2.blackbaudhosting.com


    Tracing route to a347.srip1.akasrip.net.e80e803f.1.cn.akasripcn.net [184.86.96.237]

    over a maximum of 30 hops:


      1    10 ms     8 ms     2 ms  192.168.1.1

      2    18 ms    18 ms    19 ms  96.120.11.145

      3    12 ms    12 ms    14 ms  162.151.75.177

      4    25 ms    17 ms    18 ms  ae-53-0-ar01.capitolhghts.md.bad.comcast.net [68.86.204.217]

      5    18 ms    28 ms    27 ms  be-33657-cr02.ashburn.va.ibone.comcast.net [68.86.90.57]

      6    22 ms    24 ms    26 ms  hu-0-11-0-5-pe07.ashburn.va.ibone.comcast.net [68.86.85.102]

      7    21 ms    26 ms    49 ms  50.248.116.206

      8    13 ms    20 ms    14 ms  ae0-xcr1.ash.cw.net [195.2.30.45]

      9    28 ms    27 ms    17 ms  63.130.81.84

     10    23 ms    31 ms    27 ms  208-46-117-243.dia.static.qwest.net [208.46.117.243]

     11    19 ms    21 ms    37 ms  dcx2-edge-01.inet.qwest.net [63.148.78.1]

     12    18 ms    19 ms    26 ms  dcp-brdr-03.inet.qwest.net [205.171.251.6]

     13    15 ms    14 ms    29 ms  208.173.158.13

     14    32 ms    36 ms    28 ms  cr1-tengig0-7-2-0.washington.savvis.net [204.70.197.242]

     15    25 ms    29 ms    29 ms  cr2-xe-1-0-1.jfk2.savvis.net [206.28.101.1]

     16    33 ms    38 ms    36 ms  msr1-te-0-3-0-0.bos.savvis.net [206.28.97.205]

     17    27 ms    26 ms    27 ms  hr1-te-7-2.bo3.savvis.net [206.28.97.198]
     18   217 ms   227 ms   212 ms  das2-v3009.bo3.savvis.net [205.139.96.66]

     19    35 ms    48 ms    30 ms  a184-86-96-237.deploy.static.akamaitechnologies.com [184.86.96.237]

  • If there are other folks that would like to help out, simply paste your tracert results into this thread from KB42699 posted by Jill. This is what John did to paste in his results.

    Open Command Prompt and run the following command based on your datacenter location:
    • Boston: tracert ctx-login2.blackbaudhosting.com
    • Orange County: tracert login1.blackbaudhosting.com
  • how can I post my results to that thread? I couldn't find a link.....
  • Jennifer Lucarelli:

    how can I post my results to that thread? I couldn't find a link.....


    C:\\> tracert ctx-login2.blackbaudhosting.com


    Tracing route to a347.srip1.akasrip.net.e80e803f.1.cn.akasripcn.net [157.238.70.237]

    over a maximum of 30 hops:


      1     1 ms     1 ms     1 ms  172.16.10.250

      2     *        *        *     Request timed out.

      3     *        *        *     Request timed out.

      4     *        *        *     Request timed out.

      5     *        *        *     Request timed out.

      6     *        *        *     Request timed out.

      7     *        *        *     Request timed out.

      8     *        *        *     Request timed out.

      9     *        *        *     Request timed out.

     10     *        *        *     Request timed out.

     11     *        *        *     Request timed out.

     12     *        *        *     Request timed out.

     13     *        *        *     Request timed out.

     14     *        *        *     Request timed out.

     15     *        *        *     Request timed out.

     16     *        *        *     Request timed out.

     17     *        *        *     Request timed out.

     18     *        *        *     Request timed out.

     19     *        *        *     Request timed out.

     20     *        *        *     Request timed out.

     21     *        *        *     Request timed out.

     22     *        *        *     Request timed out.

     23     *        *        *     Request timed out.

     24     *        *        *     Request timed out.

     25    58 ms    57 ms    58 ms  157-238-70-237.il.verio.net [157.238.70.237]


    Trace complete.

    Download Speed: 47317 kbps (5914.6 KB/sec transfer rate)
    Upload Speed: 6105 kbps (763.1 KB/sec transfer rate)

     

  • Jennifer Lucarelli:

     

    Jennifer Lucarelli:

    how can I post my results to that thread? I couldn't find a link.....


    C:\\> tracert ctx-login2.blackbaudhosting.com


    Tracing route to a347.srip1.akasrip.net.e80e803f.1.cn.akasripcn.net [157.238.70.237]

    over a maximum of 30 hops:


      1     1 ms     1 ms     1 ms  172.16.10.250

      2     *        *        *     Request timed out.

      3     *        *        *     Request timed out.

      4     *        *        *     Request timed out.

      5     *        *        *     Request timed out.

      6     *        *        *     Request timed out.

      7     *        *        *     Request timed out.

      8     *        *        *     Request timed out.

      9     *        *        *     Request timed out.

     10     *        *        *     Request timed out.

     11     *        *        *     Request timed out.

     12     *        *        *     Request timed out.

     13     *        *        *     Request timed out.

     14     *        *        *     Request timed out.

     15     *        *        *     Request timed out.

     16     *        *        *     Request timed out.

     17     *        *        *     Request timed out.

     18     *        *        *     Request timed out.

     19     *        *        *     Request timed out.

     20     *        *        *     Request timed out.

     21     *        *        *     Request timed out.

     22     *        *        *     Request timed out.

     23     *        *        *     Request timed out.

     24     *        *        *     Request timed out.

     25    58 ms    57 ms    58 ms  157-238-70-237.il.verio.net [157.238.70.237]


    Trace complete.

    Download Speed: 47317 kbps (5914.6 KB/sec transfer rate)
    Upload Speed: 6105 kbps (763.1 KB/sec transfer rate)

     



     

  • Lynn Slussar:

    Thanks for posting your results. I too have reopened my case from last week after reading these posts. Blackbaud told me they did not have any current issues with latency in the Raiser's Edge for hosted clients. They are now looking further into the issue as we have done all the testing

    for issues on our end. We also have RE NXT and I find it interesting that the problem could be related to having both.

    Jennifer Lucarelli:

     

    Jennifer Lucarelli:

    how can I post my results to that thread? I couldn't find a link.....


    C:\\> tracert ctx-login2.blackbaudhosting.com


    Tracing route to a347.srip1.akasrip.net.e80e803f.1.cn.akasripcn.net [157.238.70.237]

    over a maximum of 30 hops:


      1     1 ms     1 ms     1 ms  172.16.10.250

      2     *        *        *     Request timed out.

      3     *        *        *     Request timed out.

      4     *        *        *     Request timed out.

      5     *        *        *     Request timed out.

      6     *        *        *     Request timed out.

      7     *        *        *     Request timed out.

      8     *        *        *     Request timed out.

      9     *        *        *     Request timed out.

     10     *        *        *     Request timed out.

     11     *        *        *     Request timed out.

     12     *        *        *     Request timed out.

     13     *        *        *     Request timed out.

     14     *        *        *     Request timed out.

     15     *        *        *     Request timed out.

     16     *        *        *     Request timed out.

     17     *        *        *     Request timed out.

     18     *        *        *     Request timed out.

     19     *        *        *     Request timed out.

     20     *        *        *     Request timed out.

     21     *        *        *     Request timed out.

     22     *        *        *     Request timed out.

     23     *        *        *     Request timed out.

     24     *        *        *     Request timed out.

     25    58 ms    57 ms    58 ms  157-238-70-237.il.verio.net [157.238.70.237]


    Trace complete.

    Download Speed: 47317 kbps (5914.6 KB/sec transfer rate)
    Upload Speed: 6105 kbps (763.1 KB/sec transfer rate)

     

     

     

     

     

  • Lynn Slussar: We also have RE NXT and I find it interesting that the problem could be related to having both


    Hi Lynn- This is the first I'm hearing that having both products (NXT and database view) may be a contributing factor in latencies. Did a support person tell you this?
  • One month later. They've made "some changes" and the root cause is still unidentified.


     
    Hi Aaron!

    This update is in reference to your ongoing performance issues.

    While working to isolate a root cause, we have made some changes that may have improved your experience in Blackbaud Hosting. While these changes are not a solution nor has a root cause been identified at this point we would to gauge your current experience as we continue to work toward a solution.

  • We have also been experiencing slow down the past month.  The past few days was really bad so I started to see which server I was on.  Every time I was having issues, it was the same server.  I asked around the office and no one else was having issues.  So I checked to see what server they were on and none of them were the one I kept getting on.  I would log off and log in until I didn't get that particular server.  So finally today after about ten tries I got on a different server and it's been fine.  I reported the server to Blackbaud.  But it's something you might want to check.  The server I kept having issues on is D2PCXFRMTMP17.  Could be a coincidence, but I have't had any issues all day!  


    Patti Posey

    Stamford Hospital Foundation
  • Aaron Rothberg:
    I've just opened another case (13507050) for latencies this morning. I made it clear that the latencies have persisted for months, that all database view users are affected, and that all other network applications in our office do not experience latencies. Here are a few of the responses I got back from support.

    "The benefit of moving to NXT webview is that you have the cloud and everything runs as fast as your internet connection"

    "Unfortunately, with hosting, there can be latencies based on the number of people logged in at the time. We do our best to prevent any delay. "

    "there haven't been any issues listedfor this week on the Boston servor"

    "If needd, you can send a pingplotter to us using step 5 to see if there is a congestion somewhere that we can rectify."


    I'm not able to run Pingplotter for 24 hours without locking my computer (is there anyone still out there that does not have an IT policy that auto-locks your computer after a period of inactivity?), but I don't think I need the 24 hours of data. The latencies are clearly happening at the hop just before I connnect to the data center.


    329ad2b5cb1cf8495185854cff29d641-huge-sc

    7a9eabefd4850c14ef4b4b9328e5c999-huge-sc

    0ebce4063c0ea2e26a19930eb1b623eb-huge-sc



     

    Aaron,  thanks for the update.  We also have an automated process in place that locks down our PCs after so many hours of inactivity from the user.  I appreciate the details about using Pingplotter to log activity because we are experiencing slowness at times that have affected all of our users and not just the 'one off' slowness which seems to resolve itself after disconnecting and reconnecting to Citrix or a PC reboot.  Late mornings and early afternoons (11 a.m. - 2:30 p.m.) seems to be the typical slow times that we are experiencing with our hosted connection to the Boston data center.  I've reported this multiple times but I haven't pushed support for testing etc yet.  I will the next time we experience issues that affect all of our users.    Have a good day,   Annette

Categories