Serving the Quantitative Finance Community

 
cdsharm75
Topic Author
Posts: 26
Joined: November 5th, 2020, 6:40 pm
Contact:

Python: lambda functions.

September 4th, 2023, 2:25 am

I'm trying to find an easy way of calling a UDF in a lambda function in dataframe: 
Here's the dataframe:
df = pd.DataFrame({'A': ['foo', 'bar', 'baz', 'foo'], 
                   'B': ['qux', 'quux', 'quuz', 'xyz']})
df
Now the objective is to search for all rows in both columns that meet a specific criteria (user defined) and create two new columns based on that. The first new col will have a boolean value, and the second column will have a specific text value.
First: here's what works: I write the functions directly and they work. Col 2 and 3 both have the correct boolean values; based on the search conditions, only the first row returns True, all the other rows are False.
df['col2'] = (df['A'] == 'foo') & (df['B'] == 'qux' )

df = df.assign(col3 = lambda x: (df['A'] == 'foo') & (df['B'] == 'qux' ))

Of course, in the real world dataset, conditions become more convoluted. It's easier to have a separate UDF listing all the combos and then call it separately. To that end, I wrote the UDF:
def test(df):
    if (  (df['A'] == 'foo') & (df['B'] == 'qux' ) ).any():
        x = True
    else:
        x = False
    return x 

and then called it two different ways.

df = df.assign(col4 = lambda x: test(df))
# df['col5'] = df.apply(test, axis=1)
df['col6'] = df.apply(lambda x: test(df), axis=1)


As you can see from the output, the function spits out True for all rows - clearly wrong. I suspected that maybe the "any()" operator is creating some problems.....but if I remove that...I get the "...truth value of a series is ambiguous..." error. 

Would appreciate any suggestions!!
 
User avatar
katastrofa
Posts: 7185
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: Python: lambda functions.

September 5th, 2023, 12:26 pm

Why not simply
def test(df):
    x =  (df['A'] == 'foo') & (df['B'] == 'qux' ) 
    return x 
??
 
cdsharm75
Topic Author
Posts: 26
Joined: November 5th, 2020, 6:40 pm
Contact:

Re: Python: lambda functions.

September 7th, 2023, 3:02 am

Thanks! 
Took me a while to figure this out - but  it looks like there's a very specific way to call the lambda function and this is where I was going all wrong. 
Using the code you provided (and modified for a more general case), I ran the following:

def test5(df):
    if  (df['A'] == 'foo') & (df['B'] == 'qux' ):
        x = 'peak'
    elif (df['A'] == 'bar') & (df['B'] == 'quux' ):
        x = 'off'
    else:
        x = 'false'
    return x    

# this works - have to use apply not assign.
df['col10'] = df.apply(lambda x: test5(x), axis=1)

# this fails "truth value of series...." error; assign doesn't work.
df = df.assign(col11 = lambda x: test5(x))
So it took me a while to figure out that I need to use apply and not assign. I found that the assign method gave no errors when I modified the code as follows:

def test6(df):
    if ( (df['A'] == 'foo').any() & (df['B'] == 'qux' ).any() ):
        x = 'peak'
    elif ( (df['A'] == 'bar').any() & (df['B'] == 'quux' ).any() ):
        x = 'off'
    else:
        x = 'false'
    return x 

# this "works" - code doesn't fail, but has the wrong answer

df = df.assign(col12 = lambda x: test6(x))

But of course now, it doesn't give me the correct result - all rows are marked with the same label (and the use of "any" is the issue....while it prevents the "...truth value of series is ambiguous error...", it produces the wrong output).
So after a lot of googling, I figured your code combined with the correct way to call the lambda function worked. But it's been a terribly confusing and annoying journey so far....I'm glad it's over! Thanks for the help again.
 
User avatar
katastrofa
Posts: 7185
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: Python: lambda functions.

September 8th, 2023, 2:45 pm

The only difference between the two methods is that apply will modify the column on which you call it, and assign will add a new column.
I think you might need to check .any() does what you want. It checks if *any* element in the whole column is equal to the string. If you want to check each row separately, skip any() as I suggested above. But I only skimmed your posts, so I may not understand what you're doing :-D
 
cdsharm75
Topic Author
Posts: 26
Joined: November 5th, 2020, 6:40 pm
Contact:

Re: Python: lambda functions.

September 9th, 2023, 12:00 am

Thanks sir - any AND the way I was calling Lambda was creating the issue. But your post was super helpful - I had been staring at my own scribbles for so long that I just couldn't spot the issues. Again - appreciate the time!
 
User avatar
katastrofa
Posts: 7185
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: Python: lambda functions.

September 11th, 2023, 1:52 pm

No problem.
You might want to try using Copilot or even ChatGPT to get help with such issues, BTW.
 
cdsharm75
Topic Author
Posts: 26
Joined: November 5th, 2020, 6:40 pm
Contact:

Re: Python: lambda functions.

September 12th, 2023, 1:49 am

ah....totally forgot about that......will keep in mind for next time. Thanks!
 
User avatar
jasonbell
Posts: 8
Joined: May 6th, 2022, 4:16 pm
Location: Limavady, NI, UK
Contact:

Re: Python: lambda functions.

September 13th, 2023, 8:39 am

Python 3.10 now has match/case functionality. A lot neater and easier to maintain that a raft of if/then/else statements. 

Personally I'd be careful with that & within the if statement, using "and" is safer. 
Founder of Synthetica Data - https://www.syntheticadata.com
Twitter: @jasonbelldata
Author of Machine Learning: Hands on for Developers and Technical Professionals (Wiley).
 
User avatar
tagoma
Posts: 2901
Joined: February 21st, 2010, 12:58 pm

Re: Python: lambda functions.

September 15th, 2023, 9:53 am

Python 3.10 now has match/case functionality. A lot neater and easier to maintain that a raft of if/then/else statements. 

Personally I'd be careful with that & within the if statement, using "and" is safer. 


Hi JB. Wha is AND safer than & sign, please?
 
User avatar
katastrofa
Posts: 7185
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: Python: lambda functions.

September 15th, 2023, 2:00 pm

Bitwise operators don’t short-circuit
Image
 
User avatar
jasonbell
Posts: 8
Joined: May 6th, 2022, 4:16 pm
Location: Limavady, NI, UK
Contact:

Re: Python: lambda functions.

September 16th, 2023, 11:12 am

Python 3.10 now has match/case functionality. A lot neater and easier to maintain that a raft of if/then/else statements. 

Personally I'd be careful with that & within the if statement, using "and" is safer. 
Python 3.11.5 (main, Aug 24 2023, 15:08:51) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> a = 200
>>> b = 33
>>> c = 500
>>> if a > b and c > a:
...   print("Both conditions are True")
... else:
...   print("This is false!")
...
Both conditions are True

>>> a = 200
>>> b = 33
>>> c = 500
>>> if a > b & c > a:
...   print("Both conditions are True")
... else:
...   print("This is false!")
...
This is false!
>>>

^^^^^ :)
Hi JB. Wha is AND safer than & sign, please?
Hi Tagoma, straight from the CLI. And we're expected to trust sklearn ;) 
Founder of Synthetica Data - https://www.syntheticadata.com
Twitter: @jasonbelldata
Author of Machine Learning: Hands on for Developers and Technical Professionals (Wiley).
 
User avatar
tagoma
Posts: 2901
Joined: February 21st, 2010, 12:58 pm

Re: Python: lambda functions.

September 16th, 2023, 11:40 am

bitwise vs boolean. thanks guys.
no animal was hurt.
 
User avatar
katastrofa
Posts: 7185
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: Python: lambda functions.

September 20th, 2023, 12:30 am

Operator precedence above is one basic rule to remember, and here's an example of tragic consequences of using bitwise operators in short-circuiting that I mentioned before:
def cook1(recipe):
    if "salt" in recipe:
        print("Add salt.")
        return False
    else:
        return True


def cook2(recipe):
    if "salt" in recipe:
        print("Add salt.")
        return False
    else:
        return True


recipe = ["salt", "flour", "water"]

print("Bitwise AND:")
if cook1(recipe) & cook2(recipe):
    print("Voila!")

print("\nLogical AND:")
if cook1(recipe) and cook2(recipe):
    print("Voila!")
Bitwise AND:
Add salt.
Add salt.
Logical AND:
Add salt.

Bitwise cooks added twice as much salt! :(

How low have we fallen. We are discussing the basics of Python... In the past we argued about accuracy and precision of automatic differentiation vs complex step method, alternatives to modules for encapsulation in C++, non-determinism of monads in the context of Ancient Greek atomism, ... All those moments will be lost in time, like bits in NADN gate. Time to sleep.
 
User avatar
Cuchulainn
Posts: 19117
Joined: July 16th, 2004, 7:38 am
Location: 89 19 79 15

Re: Python: lambda functions.

September 24th, 2023, 7:47 pm

 
User avatar
katastrofa
Posts: 7185
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: Python: lambda functions.

September 25th, 2023, 8:58 am

A bit of an overkill in general, imho, since short circuiting in coding is used for efficiency.