Thycotic Development Blogs

The History of Searching in Secret Server

secret.server — Sun, 21 Sep 2008 07:41:00 GMT

In the recent month, we've had a lot of questions about how searching works in Secret Server, so I thought now would be a good time to answer as many questions about searching as possible.

Searching pre 5.0

Before the 5.0 edition of Secret Server searching was fairly limited. The only thing you could search on was the Secret's name. Over time, the Search criteria grew a little, but still this main limitation was always there. As soon as you wanted to search on the actual values in the secrets, you were out of luck. The ability to search by the values in a secret was one of our most requested features.

Technical Limitations

The Secret Server development team has always had a keen sense to what customers wanted, and we typically implement feature requests based on feedback. However, this particular feature had a lot of technical barriers to solve before it could be implemented.

The main barrier we had to deal with was the concept of Secret Server itself. Secret Server is designed to be as secure as possible, and one of the pieces of this design is full data encryption. All of the values of a secret, aside from its name, are stored in the database encrypted. This makes searching the database impossible. If we wanted to perform a search, we would have had to pull back every secret from the database, decrypt it, and then search it. This clearly wouldn't work from a performance angle, and didn't scale well.

Searching as of 5.0

We realized we wouldn't be able to do real-time searches on secrets. The barrier still remained though, how do we search secrets and not expose sensitive information? Our solution was a hash based index table.

A What?

Many systems, such as Windows and search providers like Google keep a search index. When you search Google, you really aren't searching the entire Internet all at once, you are searching a dictionary of content that Google has built over time. Secret Server does something similar. The trick is to build an index but also keep it secure.

Secret Server 5.0 has a background monitor, the Search Indexer, that looks for changed secrets, about every 60 seconds it queries the database looking for unindexed secrets or changes in secrets. When you create or modify a secret, we flag that secret to tell the Search Indexer to re-index it.

Security

The Search Indexer creates hashed terms from the values in a secret. More specifically for those technically interested, we use the HMAC-512 algorithm. A quick explanation of what this algorithm does is creates a one-way code. For example, if the word "book" was hashed, it would produce a unique output. However this output cannot be converted back into the original data, "book" in our case.

This technique is used when creating indexes. Let's say we have a secret with a field called "Server" with a value of "OFFICE\Webserver01". When the search indexer got around to indexing this secret, it would create a hashed value of "office\webserver01". Whenever we create hashed terms, we always convert it to lowercase so that searching isn't case-sensitive. This search index record would become associated with the secret.

Now, when a user does a search, we use the same hash algorithm to compute the hash term of what you are searching for (again converted to lowercase). We when search our index table for a match.

What About Partial Matches?

When we have a term like "OFFICE\Webserver01", we produce hashes of "pieces" of the word. In this case, we would also produce specific hashes for "office" and "webserver01". Notice that we split on the letter "\". The same happens when a search is performed. This way if you searched for "OFFICE\Webserver02", it would still come back with the OFFICE\Webserver01" because the "OFFICE" term still matched. We do this for other letters as well, that includes spaces, backslashes, slashes, periods, commas, and semicolons.

Search Index Modes

The Search Indexer has two modes. Standard, and Enhanced. So far, all of the behavior I have described has been the "Standard" mode. The Enhanced mode works very similarly, however it also produces three letter partials. Using out "OFFICE\Webserver01" example, we produce our hashes normally, but we also produce the partials. We would add hashes for "OFF", "FFI", "FIC", "ICE", etc. This allows partial matches to return.

So Many Results

The implementation sounds correct, but it has some room for improvement. Note that I said we split the terms on periods. That means if you searched for "foo@test.com", it would return everything that had "com" in it, and chances are there are a lot of results. The splitting on the period seems to be the biggest culprit for undesired results coming back. Once you throw the Enhanced mode into the mix, it gets even more complicated.

Looking Forward

Nothing has been set in stone in terms of changes and when it will be implemented, but we have been kicking around a lot of ideas. The immediate one might be to consider removing the period from the characters that we split on. Another idea was ranking the results. Secret Server right now always returns secrets sorted by their name. It would make more sense if we returned results in order of the number of hash terms that matched and if the name matched as well.

I hope that clarifies some of the mystery surrounding search. If you have any additional feedback or questions, be sure to drop by our forums and let us know!

-- Kevin

Sneak Peek: PuTTY Launcher

secret.server — Thu, 11 Sep 2008 21:52:53 GMT

One of a system administrator's must-have items in his toolbox is PuTTY. PuTTY is a small, lightweight program that is perfect for telnet and SSH connections. It doesn't require any installation, it's just a single EXE file and you're good to go.

A feature of Secret Server that I personally have always found extremely useful is the launching capability that we introduced with Remote Desktop. It's very handy for starting Remote Desktop sessions. We decided to take it a step further and extend this functionality to PuTTY.

An initial obstacle that needed to be overcome was figuring out how to make sure PuTTY was on the client's machine. The creators of PuTTY are generous, and fortunately they allow us to distribute PuTTY with Secret Server. Since the Remote Launcher capability is a Microsoft ClickOnce application, it seemed reasonable to distribute PuTTY with our application. This would avoid the need for users having to tell our application where to look for PuTTY, or us requiring that you have it in a certain location on the machine.

However, PuTTY is 500 kilobytes, and the initial application was a mere 12 kilobytes. 500K is small in today's high tech world, but to reduce corporate bandwidth use, we only distribute it when you need it for the first time. That means when you make your first launch of PuTTY, we'll download the application for you from your Secret Server installation, thus not needing an outside Internet connection, but after that it's cached so you only need to download it once.

Once PuTTY is downloaded successfully, the application will automatically start already logged in at the prompt. For the first release of the PuTTY launcher, we will only support SSH.

If you want to see additional launchers built into Secret Server, make sure you stop by our forums and let us know!

-- Kevin

Sneak Peek - Secret Server 5.0 and Searching Fields

secret.server — Mon, 07 Jul 2008 03:05:52 GMT

Secret Server 5.0 is currently under development, and one of the features that we know for sure that will be in 5.0 is searching Secret Fields. This has been a popular request. We had several obstacles to achieve this, and we have implemented a solution that is secure but effective.

The search works by Secret Server creating an index catalog for search terms for each and every secret. This runs as a background process. Secret Server will then start indexing all existing Secrets in your installation, and maintain indexes for secrets as they are changed.

The indexing service allows two different modes of indexing. The standard mode, which allows you to search on whole words. The Extended Indexing option allows searching on part of a word with a precision of 3 characters. For example, "sec" would make a field with the value of "Secret", as would "secre".

Stay tuned for more features coming in Secret Server 5.0!

-- Kevin

Text Encoding (Part-1)

vcsjones — Mon, 19 May 2008 10:57:15 GMT

I was recently on the ASP.NET Forums and a member was asking, "How can I figure out the encoding of text?" and that got me thinking. There should be a reasonable way to do this, right? It's a useful thing to know. First, we need a little Read More......(read more)

Why does Secret Server take so long to start up?

secret.server — Sun, 18 May 2008 23:14:20 GMT

One of the things that we did notice with Secret Server is that it does take what seems to be a long time for Secret Server to start up for the first time. This started happening in Secret Server 4.0. So, what exactly is going on?

Secret Server does some startup tasks for the first time. Namely, it starts up some background monitoring tasks for synchronizing Active Directory and the Remote Password changing features. There is one more though that takes up most of the time, and that is verifying all of the Strong Name signatures.

First, what is a Strong Name? When we release Secret Server, we send out all of the DLLs with a digital signature on all of the assemblies. Secret Server has multiple DLLs that talk to each other. Now, what's stopping someone with access to the server from dropping in a fake DLL that looks like ours, but it is also secretly emailing out information? Step in strong names. When the .NET Framework loads all of the assemblies for a particular application, it ensure that all of the assemblies have the strong name key that was used when it was compiled. If the Strong Name keys don't match, then the .NET Framework won't accept it. Since only Thycotic has the key, it cannot be faked.

This is a somewhat lengthy process for the .NET Framework, as it will also have to calculate checksums of the entire assembly as well. Not to mention that this entire process occurs for all 14 of the assemblies in Secret Server.

-- Kevin

Too Little Too Late?

vcsjones — Tue, 13 May 2008 20:50:23 GMT

I booted up my PC today and saw this nice message from Adobe telling me that there was an update for my Flash installation. I couldn't help but notice that one of the highlighted features was support for HD content. I can't help but feel that Read More......(read more)

CMAP User Group Presentation

vcsjones — Mon, 05 May 2008 11:23:45 GMT

On Tuesday, May 6 th at the CMAP User Group Meeting will be Heroes Happen {here} Launch where I will be discussing the new features in SQL 2008 and Steve Michelotti will be discussing the new C# 3.0 Language Enhancements. If you are in the Baltimore area, Read More......(read more)

Secret Server on the Treo 700

secret.server — Sat, 19 Apr 2008 12:25:06 GMT

Secret Server has supported a "Mobile Edition" for over a year now but it is always tricky making sure that it works correctly on all devices.

Our approach was to bake mobile support into the base product (ASP.NET based) so it simply scales down to the capability of the device. That sounds simple but unfortunately it depends on making sure that functionality will work with all the limitations of various devices.

My own favorite BlackBerry 8820 does a reasonable job of helping me get to the password I need in emergencies but it is hardly a pleasant browsing experience. In fairness, no browsing on the device is particularly pleasant since it is slow, struggles with most layouts and has a small screen. That said, I love it dearly and browsing has never been a core requirement for me since email, contacts and calendar are definitely my most essential.

Today we had a customer ask about the Treo 700 so I tried out the emulator from the Palm website. It seems to work fine with Secret Server and I was able to browse around and access passwords.

--Jonathan

Secret Server on the Treo 700

Jonathan Cogley's Blog — Fri, 18 Apr 2008 18:47:00 GMT

Secret Server has supported a "Mobile Edition" for over a year now but it is always tricky making sure that it works correctly on all devices. Our approach was to bake mobile support into the base product (ASP.NET based) so it simply scales Read More......(read more)

CMAP Code Camp

vcsjones — Sun, 13 Apr 2008 12:41:13 GMT

Saturday was the Spring '08 CMAP Code Camp. Lots of good sessions there, lots of fun as well! I presented on "SQL Server 2008" and "What's new in C# 3.0" as a replacement for Jay Flowers since he wasn't able to make it. Read More......(read more)

A sign that you are no longer a startup.

Jonathan Cogley's Blog — Thu, 10 Apr 2008 23:20:09 GMT

Ok, so it is a really bad pun. It is tough writing blog post titles sometimes. :) Anyway, we finally got a beautiful metallic official sign for our office after inhabiting our office space for almost two years. (Now if we could just do something Read More......(read more)

Code Camps this weekend!

Jonathan Cogley's Blog — Thu, 10 Apr 2008 22:54:04 GMT

There are two code camps this weekend: CMAP Code Camp in Maryland Pittsburgh Code Camp in Pennsylvania I will be speaking at the Pittsburgh Code Camp on Refactoring - a topic that is very dear to me. Register now and come along to talk code. We Read More......(read more)

I'd like to report a negligence

vcsjones — Tue, 08 Apr 2008 22:50:58 GMT

I've always been interested in software security, and it's always been a number one priority for me. Software security is really honoring the trust of the people that use your software. I've also been fortunate to be the lead developer of Read More......(read more)

MVC Framework

vcsjones — Tue, 08 Apr 2008 00:33:11 GMT

The MVC framework is the new "hot" thing in the ASP.NET world for developers. As such, everyone has at least one blog entry about it. So, I think it's time I jumped on that ship. Though, I wanted to voice a few concerns with the MVC Framework, Read More......(read more)

When you apply for a job, read the job posting!

Jonathan Cogley's Blog — Mon, 07 Apr 2008 22:48:00 GMT

I am constantly amazed by the number of job candidates who apply for a position without reading the job posting. We are currently hiring for three positions: Agile .NET Developer Agile .NET Developer Internship Graphics Designer All three involve *doing* Read More......(read more)

Blog Moving

vcsjones — Sun, 06 Apr 2008 14:42:00 GMT

Hi everyone, I've decided to move my blog to http://msmvps.com/blogs/vcsjones , so please update booksmarks and RSS feeds. However, I will continue to have my new blog cross-post on this site as well. Share this post: email it! | bookmark it! | digg Read More......(read more)

CaptainHook

vcsjones — Sun, 06 Apr 2008 14:35:00 GMT

This is my first post to the msmvps.com web site. I decided to move my blog from http://community.strongcoders.com/blogs/vcsjones . So, I thought I would start off by talking about something a little obscure that a co-worker turned me onto. OK, different Read More......(read more)

Secret Server at FOSE 2008

Jonathan Cogley's Blog — Thu, 03 Apr 2008 21:59:57 GMT

We took the Secret Server booth to the FOSE 2008 Conference this week. FOSE is the largest IT event for US Government. It wasn't really a very long trip for us ... about 12 blocks east from our offices in Dupont Circle in Washington DC to Read More......(read more)

Secret Server at FOSE 2008

secret.server — Thu, 03 Apr 2008 15:27:22 GMT

This year Secret Server made its debut at FOSE, one of the leading government technology events in the nation. The show is being held at the Walter E. Washington Convention Center which is situated only a few minutes away from our offices in downtown D.C.

Despite there being several hundred kiosks and lectures, Secret Server appears to be one of the few software products featured. Many of the exhibitions are displaying hardware and energy saving innovations. I think a lot of people have been pleasantly surprised to see a solution for password management.

Over the last couple of days, I and some of the other team members got a chance to interact with attendees and demonstrate some of the core functionality of Secret Server. We have received a lot of enthusiasm and great feedback on the product thus far.

Today is the final day for FOSE. Come visit us at booth #100 located in the Security section. Hope to see you there!

--Joseph

April Fools

vcsjones — Tue, 01 Apr 2008 10:49:02 GMT

Yes, April Fools - one of my favorite days. What April Fool's day is not complete without covering your co-worker's desk and chair in sticky notes? Though as a side note, I was re-awarded for ASP.NET MVP. Share this post: email it! | bookmark Read More......(read more)

Next Richmond Code Camp

vcsjones — Sat, 29 Mar 2008 19:16:01 GMT

It's On! The next Richmond Code Camp is set for Saturday, April 26 th . Every year they keep getting better and better, so this one is going to be the best yet! I'm set to speak there about some of the new stuff in SQL 2008 for Developers and Read More......(read more)

Don't miss the Nova Code Camp South this weekend!

Jonathan Cogley's Blog — Fri, 28 Mar 2008 08:34:07 GMT

The NoVa CodeCamp South v1 will be held on March 29th 2008 in Woodbridge VA. The speaker schedule has been posted here . I am presenting two sessions: 9:00-10:15: Refactoring in C# 1:00-2:15: Read More......(read more)

Giving Secret View a System Font

secret.server — Thu, 27 Mar 2008 23:54:47 GMT

One of the questions that I sometimes get from customers is, "I want the information on the Secret View page to display in a system font". The reason for this is it makes it easier to distinguish between O's and zeros; and lower-case L's and capital I's.

This can easily be accomplished with CSS, and with since Secret Server 4.0 and up supports Themes, it is simple enough to add your own CSS to the default.css file.

Because of the way Copy to Clipboard works, all of the attributes that contain information are held in a custom attribute "t". This attribute is on the span elements and the text boxes when in edit mode. In theory, it should be as simple as this:

*[t]
{
font-family:Consolas,System;
}

This is part of the CSS 2 specification, and the selector states "Any element with the attribute 't'." As expected, this works well with FireFox. This took care of the labels and the text boxes all-in-one. However, IE presented a bit of an issue. This simple solution didn't seem to work. It's not a secret to web developers that Trident, IE's rendering engine, is pretty buggy as far as rendering engines go. What surprised me more was that the IE 8 beta, the up-and-coming super-compliant version of IE, still did not take. What was strange that when using a simple test page, the attribute selector did work; so it is supported in IE 7 and 8. There just appears to be an issue with that particular page.

So the solution became a little more complex. A lot of the elements on the secret view page don't have classes or ID's at the moment, which makes applying CSS to just some of the elements a bit trickier. In the end, this is how it turned out:

And the CSS used to accomplish this that works in both IE and FireFox:

div#SecretViewDialog * td.SecretFieldCell span, * span#iSM li
{
font-family:Verdana ! important;
font-size:10pt ! important;
}

input.SecretViewTextbox, input.SecretPasswordTextbox, div#SecretViewDialog * span
{
font-family:Consolas,System;
font-size:11pt;
}

The font of my choice is Consolas, a nice font that makes it easy to distinguish characters. It is a free font for user's that own Visual Studio 2005 via download, and also ships with Visual Studio 2008.

-- Kevin

Can you find the bug in this code? (THE FIX)

Jonathan Cogley's Blog — Wed, 26 Mar 2008 12:30:35 GMT

Thanks to everyone for contributing! It was really neat to read everyone's ideas and see the discussion and review (talking about code is always fun!). Here is a summary of responses and the "fixed" code. If you are interested Read More......(read more)

For once, not blogging wasn't my fault

Jonathan Cogley's Blog — Tue, 25 Mar 2008 08:09:35 GMT

For the last couple of days, the weblogs.asp.net website has been unable to accept posts from Windows Live Writer. For the first time in the history of this blog, I can blame someone else for not posting. In fact, I even started queueing up Read More......(read more)